2024-03-29T10:58:27.503 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:58:27.504 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676 --verbose -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676/orig.config.yaml 2024-03-29T10:58:27.874 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:58:27.975 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622676' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEzpVDeER3LSqzcPCX2onA/tmlitjWMAGbeI4lRfZ3+IlWdRVJ1SVubTqvC3OzjxhI6SW7/YDv4bE6CdsPrdO1s= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKZbT9vO8FBP38Ohxdh8GmmMUosHlrzHzM47LuGueymclBqiORtuRYYEGfp1HyRqIjnEX2vis1T6Lk4LPFAsLGA= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:58:27.975 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:58:27.976 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:58:27.977 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:58:27.979 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:58:28.016 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:58:28.016 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:58:28.016 INFO:teuthology.packaging:ref: None 2024-03-29T10:58:28.016 INFO:teuthology.packaging:tag: None 2024-03-29T10:58:28.016 INFO:teuthology.packaging:branch: main 2024-03-29T10:58:28.016 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:58:28.017 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:58:28.384 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:58:28.386 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:58:28.391 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:58:28.391 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:58:28.400 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:58:28.416 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:58:28.421 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:58:28.450 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:52:49.649025', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEzpVDeER3LSqzcPCX2onA/tmlitjWMAGbeI4lRfZ3+IlWdRVJ1SVubTqvC3OzjxhI6SW7/YDv4bE6CdsPrdO1s='} 2024-03-29T10:58:28.475 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:52:49.650491', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKZbT9vO8FBP38Ohxdh8GmmMUosHlrzHzM47LuGueymclBqiORtuRYYEGfp1HyRqIjnEX2vis1T6Lk4LPFAsLGA='} 2024-03-29T10:58:28.475 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:58:28.483 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:58:28.483 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:58:28.483 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:58:28.555 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f436b0f90d0>, signals=[15]) 2024-03-29T10:58:28.555 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:58:28.563 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:58:28.563 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T10:58:28.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:58:28.659 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T10:58:28.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:58:28.747 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:58:28.754 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-03-29T10:58:28.772 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-03-29T10:58:28.773 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-03-29T10:58:28.830 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2024-03-29T10:58:28.830 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="9" 2024-03-29T10:58:28.830 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2024-03-29T10:58:28.830 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:58:28.830 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="9" 2024-03-29T10:58:28.830 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:58:28.830 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:58:28.831 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2024-03-29T10:58:28.831 INFO:teuthology.orchestra.run.smithi157.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:58:28.831 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:58:28.831 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:58:28.831 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:58:28.831 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:58:28.831 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:58:28.832 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-03-29T10:58:28.861 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-03-29T10:58:28.878 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-03-29T10:58:28.878 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-03-29T10:58:28.935 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-03-29T10:58:28.935 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-03-29T10:58:28.935 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-03-29T10:58:28.935 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:58:28.935 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-03-29T10:58:28.935 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:58:28.935 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:58:28.935 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-03-29T10:58:28.935 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:58:28.935 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:58:28.936 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:58:28.936 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:58:28.936 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:58:28.936 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:58:28.936 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-03-29T10:58:28.961 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:58:28.974 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:58:28.979 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:58:28.979 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:58:28.982 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:58:28.997 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:58:29.003 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:58:29.003 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:58:29.039 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:58:29.124 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:58:29.181 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:58:29.196 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:58:29.197 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:58:29.197 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:58:29.197 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:58:29.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:58:29.214 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-03-29T10:58:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:58:29.215 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-03-29T10:58:29.270 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:58:29.270 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.14.0-361.el9.x86_64 2024-03-29T10:58:29.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2024-03-29T10:58:29.271 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:58:29.271 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.14.0-361.el9.x86_64 2024-03-29T10:58:29.272 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2024-03-29T10:58:31.346 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T10:58:31.471 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T10:58:32.964 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-29T10:58:33.088 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-29T10:58:36.920 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-29T10:58:39.763 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-29T10:58:41.953 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 952 kB/s | 8.0 MB 00:08 2024-03-29T10:58:43.929 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-29T10:58:43.955 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 6.1 MB 00:00 2024-03-29T10:58:45.272 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 92 kB/s | 16 kB 00:00 2024-03-29T10:58:47.940 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 21 MB 00:02 2024-03-29T10:58:50.637 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.1 MB 00:03 2024-03-29T10:58:52.247 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-29T10:58:53.140 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 5.9 kB/s | 1.7 kB 00:00 2024-03-29T10:58:53.875 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T10:58:55.396 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:58:55.397 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:58:55.498 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T10:58:55.504 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T10:58:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T10:58:55.504 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T10:58:55.504 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T10:58:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:58:55.504 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-29T10:58:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:58:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:58:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:58:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T10:58:55.505 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T10:58:55.505 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T10:58:55.505 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-03-29T10:58:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T10:58:55.505 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 76 M 2024-03-29T10:58:55.506 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 123 M 2024-03-29T10:58:55.506 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T10:58:55.657 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T10:58:56.441 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-03-29T10:58:56.991 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 23 MB/s | 34 MB 00:01 2024-03-29T10:58:57.450 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T10:58:57.452 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:57.453 INFO:teuthology.orchestra.run.smithi157.stdout:Total 39 MB/s | 76 MB 00:01 2024-03-29T10:58:57.453 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T10:58:57.837 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T10:58:57.838 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T10:58:58.697 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T10:58:58.697 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T10:58:58.948 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-29T10:59:01.008 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T10:59:01.255 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:59:01.255 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:59:01.352 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T10:59:01.369 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T10:59:01.370 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-03-29T10:59:01.370 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T10:59:01.370 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 76 M 2024-03-29T10:59:01.370 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 123 M 2024-03-29T10:59:01.370 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T10:59:01.516 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:59:01.595 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:01.692 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:02.066 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-03-29T10:59:02.583 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T10:59:02.933 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T10:59:02.936 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:02.936 INFO:teuthology.orchestra.run.smithi195.stdout:Total 49 MB/s | 76 MB 00:01 2024-03-29T10:59:02.936 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T10:59:03.286 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T10:59:03.287 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T10:59:04.163 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T10:59:04.163 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T10:59:04.468 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:04.621 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:06.471 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T10:59:07.107 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:07.184 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:09.461 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:09.929 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:10.069 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:14.179 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:14.746 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:14.861 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:19.604 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:43.358 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:43.358 INFO:teuthology.orchestra.run.smithi157.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:59:43.358 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T10:59:43.402 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:43.777 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:43.777 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:43.777 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:43.777 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:44.174 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:44.174 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T10:59:44.174 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T10:59:44.174 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:44.174 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:59:44.175 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:59:44.175 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:59:44.175 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T10:59:44.175 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T10:59:44.505 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:59:44.606 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:44.606 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:59:44.607 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:59:44.607 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:59:44.607 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:59:44.607 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:59:44.607 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:59:44.607 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:59:44.687 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:44.688 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:59:44.688 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2024-03-29T10:59:44.732 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2024-03-29T10:59:44.732 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2024-03-29T10:59:44.732 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.06 2024-03-29T10:59:44.732 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 68.el9 2024-03-29T10:59:44.732 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2024-03-29T10:59:44.732 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:59:44.732 INFO:teuthology.orchestra.run.smithi157.stdout:Group : Unspecified 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 8264254 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : builder@centos.org 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T10:59:44.733 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:59:44.734 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:59:44.734 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:59:44.734 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2024-03-29T10:59:44.734 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T10:59:44.734 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:59:44.735 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:59:44.735 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:59:44.735 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:59:45.376 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-03-29T10:59:46.765 INFO:teuthology.orchestra.run.smithi157.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:59:46.798 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-03-29T10:59:46.800 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-29T10:59:46.815 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.t1jzTsw5CW 2024-03-29T10:59:46.815 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.t1jzTsw5CW 2024-03-29T10:59:46.884 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.t1jzTsw5CW 2024-03-29T10:59:47.028 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.t1jzTsw5CW is 6KB 2024-03-29T10:59:47.047 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.t1jzTsw5CW 2024-03-29T10:59:47.060 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:59:47.125 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:59:47.125 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:59:47.125 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:59:47.125 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:59:47.126 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:59:47.268 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-03-29T10:59:49.193 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:49.193 INFO:teuthology.orchestra.run.smithi195.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:59:49.193 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T10:59:49.236 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:49.633 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:49.633 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:49.633 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:49.633 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:50.034 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:50.034 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T10:59:50.034 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T10:59:50.034 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:50.034 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:59:50.034 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:59:50.035 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:59:50.035 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T10:59:50.035 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T10:59:50.354 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:59:50.449 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:50.449 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:59:50.449 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:59:50.449 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:59:50.449 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:59:50.449 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:59:50.449 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:59:50.449 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:59:50.529 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:50.529 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:59:50.529 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2024-03-29T10:59:50.574 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2024-03-29T10:59:50.574 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2024-03-29T10:59:50.574 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.06 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 68.el9 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Group : Unspecified 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 8264254 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : builder@centos.org 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T10:59:50.575 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:59:50.576 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:59:50.576 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:59:50.576 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2024-03-29T10:59:50.576 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T10:59:50.576 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:59:50.576 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:59:50.577 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:59:50.577 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:59:51.136 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-03-29T10:59:52.472 INFO:teuthology.orchestra.run.smithi195.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:59:52.510 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-03-29T10:59:52.512 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-03-29T10:59:52.527 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.1DyqylvkEs 2024-03-29T10:59:52.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1DyqylvkEs 2024-03-29T10:59:52.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.1DyqylvkEs 2024-03-29T10:59:52.747 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.1DyqylvkEs is 6KB 2024-03-29T10:59:52.795 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.1DyqylvkEs 2024-03-29T10:59:52.809 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:59:52.875 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:59:52.875 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:59:52.875 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:59:52.875 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:59:52.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:59:53.018 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-03-29T10:59:53.102 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-29T11:00:17.294 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:00:17.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-29T11:00:17.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:00:23.030 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:00:23.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-29T11:00:23.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:00:35.742 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-29T11:00:41.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-03-29T11:00:44.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-29T11:00:44.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:00:47.810 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-29T11:00:50.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-29T11:00:50.379 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:00:59.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-29T11:00:59.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:02.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-29T11:01:17.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-29T11:01:17.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:36.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-29T11:01:50.395 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:01:54.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-29T11:01:54.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:54.726 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-03-29T11:01:55.193 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-29T11:01:55.194 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:01:55.194 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-03-29T11:01:55.208 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:01:55.208 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:01:55.208 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:01:55.208 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:01:56.208 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:01:56.209 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:01:56.277 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-03-29T11:01:56.315 DEBUG:teuthology.parallel:result is None 2024-03-29T11:02:02.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-29T11:02:02.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:02:02.757 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-03-29T11:02:03.126 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-29T11:02:03.126 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:02:03.126 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-03-29T11:02:03.146 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:02:03.146 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:02:03.146 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:02:03.146 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:02:04.149 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:02:04.149 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:02:04.232 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-03-29T11:02:04.265 DEBUG:teuthology.parallel:result is None 2024-03-29T11:02:04.266 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:02:04.275 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:02:04.275 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:02:04.278 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:02:04.295 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:02:04.302 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:02:04.311 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:02:04.311 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:02:04.336 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:02:04.379 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:02:04.386 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:02:04.386 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:02:04.417 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:02:04.451 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:02:04.456 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:02:04.464 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:02:04.471 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:02:04.473 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:02:04.480 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:02:04.480 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:02:04.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:02:04.545 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:02:04.554 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:02:04.554 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:02:04.587 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:02:04.602 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:02:04.681 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:02:04.736 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:02:04.818 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:02:04.862 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:02:04.862 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:02:04.931 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:02:04.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:02:05.030 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:02:05.113 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:02:05.161 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:02:05.161 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:02:05.231 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-03-29T11:02:05.233 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-03-29T11:02:05.273 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:02:05.314 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:02:05.797 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:02:05.807 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:02:05.807 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:02:05.825 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:02:05.834 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2024-03-29T11:02:05.902 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2024-03-29T11:02:05.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-03-29T11:02:05.964 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-03-29T11:02:05.966 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:02:05.966 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2024-03-29T11:02:06.005 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2024-03-29T11:02:06.006 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-03-29T11:02:06.026 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-03-29T11:02:06.027 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi157.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:02:06.027 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:02:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:02:06.074 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:02:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:02:06.102 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:02:06.103 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:02:06.149 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:02:06.186 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:02:06.196 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:02:06.232 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:02:06.233 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi157.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:08:27.527 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi157.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-03-29T11:08:27.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-29T11:08:27.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:27.621 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-03-29T11:08:27.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-29T11:08:27.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-29T11:08:27.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:27.796 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-03-29T11:08:27.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-29T11:08:27.878 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:08:27.888 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:08:27.888 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:08:27.888 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:08:27.891 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:08:27.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:08:27.920 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:08:27.937 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:08:27.949 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:08:27.966 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2024-03-29T11:08:27.966 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:08:27.979 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2024-03-29T11:08:27.995 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-03-29T11:08:27.997 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:08:28.009 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-03-29T11:08:28.013 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:08:28.025 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:08:28.042 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:08:28.073 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:08:28.076 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:08:28.076 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-03-29T11:08:28.076 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:08:28.077 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:08:28.077 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:08:28.077 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:08:28.093 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:08:28.097 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:08:28.097 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-03-29T11:08:28.097 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:08:28.097 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:08:28.097 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:08:28.097 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:08:28.097 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:08:28.106 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:08:28.106 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:08:28.106 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:08:28.106 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:08:28.112 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:08:28.112 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:08:28.113 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:08:28.113 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:08:28.115 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:08:28.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:08:28.255 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:08:28.256 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:08:28.330 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:08:28.330 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:08:28.428 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:08:28.428 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:08:28.428 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:08:28.457 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:08:28.457 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:08:28.498 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:08:28.499 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:08:28.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:08:28.526 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:08:28.528 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:08:28.528 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:08:28.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:08:28.606 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:08:28.673 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2024-03-29T11:08:28.674 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2024-03-29T11:08:28.677 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:08:28.743 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2024-03-29T11:08:28.745 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2024-03-29T11:08:28.917 INFO:teuthology.orchestra.run.smithi157.stdout:74 files removed 2024-03-29T11:08:28.943 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw 2024-03-29T11:08:28.980 INFO:teuthology.orchestra.run.smithi195.stdout:74 files removed 2024-03-29T11:08:29.007 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw 2024-03-29T11:08:29.689 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 185 kB/s | 82 kB 00:00 2024-03-29T11:08:29.784 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 172 kB/s | 82 kB 00:00 2024-03-29T11:08:30.131 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-29T11:08:30.163 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-29T11:08:30.440 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 7.0 kB/s | 1.9 kB 00:00 2024-03-29T11:08:30.513 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-29T11:08:31.343 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - AppStream 7.7 MB/s | 6.7 MB 00:00 2024-03-29T11:08:31.757 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - AppStream 5.6 MB/s | 6.7 MB 00:01 2024-03-29T11:08:32.768 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-29T11:08:33.265 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-29T11:08:37.325 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-29T11:08:37.672 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:04 2024-03-29T11:08:39.390 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-29T11:08:43.658 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-29T11:08:45.458 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-29T11:08:47.277 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T11:08:52.219 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-29T11:08:54.876 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:08:54.900 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:08:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T11:08:54.900 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:08:54.900 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:08:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:08:54.900 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2024-03-29T11:08:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:08:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:08:54.900 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-29T11:08:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:08:54.901 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:08:54.902 INFO:teuthology.orchestra.run.smithi195.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:08:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:08:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:08:54.903 INFO:teuthology.orchestra.run.smithi195.stdout:Installing weak dependencies: 2024-03-29T11:08:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:08:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:08:54.903 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:08:54.903 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:08:54.903 INFO:teuthology.orchestra.run.smithi195.stdout:Install 30 Packages 2024-03-29T11:08:54.903 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 2 Packages 2024-03-29T11:08:54.903 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:08:54.904 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 58 M 2024-03-29T11:08:54.904 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:08:56.144 INFO:teuthology.orchestra.run.smithi195.stdout:(1/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 17 MB/s | 20 MB 00:01 2024-03-29T11:08:56.186 INFO:teuthology.orchestra.run.smithi195.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 608 kB/s | 25 kB 00:00 2024-03-29T11:08:56.378 INFO:teuthology.orchestra.run.smithi195.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.8 MB/s | 742 kB 00:00 2024-03-29T11:08:56.430 INFO:teuthology.orchestra.run.smithi195.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 9.8 MB/s | 498 kB 00:00 2024-03-29T11:08:56.729 INFO:teuthology.orchestra.run.smithi195.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 17 MB/s | 5.1 MB 00:00 2024-03-29T11:08:56.770 INFO:teuthology.orchestra.run.smithi195.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-29T11:08:56.821 INFO:teuthology.orchestra.run.smithi195.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-29T11:08:56.879 INFO:teuthology.orchestra.run.smithi195.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T11:08:56.946 INFO:teuthology.orchestra.run.smithi195.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 4.7 MB/s | 320 kB 00:00 2024-03-29T11:08:56.997 INFO:teuthology.orchestra.run.smithi195.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.8 MB/s | 298 kB 00:00 2024-03-29T11:08:57.064 INFO:teuthology.orchestra.run.smithi195.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 99 kB 00:00 2024-03-29T11:08:57.172 INFO:teuthology.orchestra.run.smithi195.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-29T11:08:57.223 INFO:teuthology.orchestra.run.smithi195.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T11:08:57.256 INFO:teuthology.orchestra.run.smithi195.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:08:57.306 INFO:teuthology.orchestra.run.smithi195.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-29T11:08:57.348 INFO:teuthology.orchestra.run.smithi195.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T11:08:57.391 INFO:teuthology.orchestra.run.smithi195.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 985 kB/s | 41 kB 00:00 2024-03-29T11:08:57.466 INFO:teuthology.orchestra.run.smithi195.stdout:(18/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 2.1 MB/s | 5.3 MB 00:02 2024-03-29T11:08:57.491 INFO:teuthology.orchestra.run.smithi195.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 220 kB 00:00 2024-03-29T11:08:57.517 INFO:teuthology.orchestra.run.smithi195.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 699 kB/s | 35 kB 00:00 2024-03-29T11:08:57.542 INFO:teuthology.orchestra.run.smithi195.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-29T11:08:57.659 INFO:teuthology.orchestra.run.smithi195.stdout:(22/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 3.6 MB/s | 9.7 MB 00:02 2024-03-29T11:08:57.901 INFO:teuthology.orchestra.run.smithi195.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 108 kB/s | 26 kB 00:00 2024-03-29T11:08:58.068 INFO:teuthology.orchestra.run.smithi195.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 561 kB/s | 309 kB 00:00 2024-03-29T11:08:58.093 INFO:teuthology.orchestra.run.smithi195.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-03-29T11:08:58.201 INFO:teuthology.orchestra.run.smithi195.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-03-29T11:08:58.318 INFO:teuthology.orchestra.run.smithi195.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-29T11:08:58.594 INFO:teuthology.orchestra.run.smithi195.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2024-03-29T11:08:58.852 INFO:teuthology.orchestra.run.smithi195.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 3.0 MB/s | 1.6 MB 00:00 2024-03-29T11:08:59.136 INFO:teuthology.orchestra.run.smithi195.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-29T11:08:59.203 INFO:teuthology.orchestra.run.smithi195.stdout:(31/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 4.4 MB 00:01 2024-03-29T11:08:59.345 INFO:teuthology.orchestra.run.smithi195.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.5 MB/s | 3.4 MB 00:00 2024-03-29T11:08:59.348 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:59.348 INFO:teuthology.orchestra.run.smithi195.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-29T11:08:59.349 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:08:59.387 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:08:59.387 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:08:59.859 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:08:59.859 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:09:00.556 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 874 kB/s | 19 MB 00:22 2024-03-29T11:09:01.322 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:09:01.407 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:09:01.449 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:09:01.475 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:09:01.497 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:09:01.514 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:09:01.667 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:09:01.719 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:09:02.095 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:09:02.147 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:09:02.217 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:09:02.305 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:09:02.328 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:09:02.373 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:09:02.402 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:09:02.449 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:09:02.514 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:09:02.538 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:09:02.602 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:09:02.637 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:09:02.744 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:09:02.780 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:09:02.820 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:09:02.853 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:09:02.883 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:09:02.915 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:09:03.227 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:09:03.253 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:09:03.326 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:09:03.357 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:09:03.398 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:09:03.866 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:09:03.897 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:09:03.968 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:09:04.036 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:09:04.066 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:09:04.982 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.1 MB 00:00 2024-03-29T11:09:06.143 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:09:06.187 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:09:06.350 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-03-29T11:09:06.791 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:09:06.824 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:09:06.909 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:09:06.909 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:09:06.909 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:09:06.909 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:06.950 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:09:07.634 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-29T11:09:12.651 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-29T11:09:15.230 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:09:15.248 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:09:15.248 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T11:09:15.248 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:09:15.248 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:09:15.248 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:09:15.248 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:09:15.249 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:09:15.250 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout:Installing weak dependencies: 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout:Install 30 Packages 2024-03-29T11:09:15.251 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 2 Packages 2024-03-29T11:09:15.252 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:15.252 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 58 M 2024-03-29T11:09:15.252 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:09:16.577 INFO:teuthology.orchestra.run.smithi157.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 5.7 MB/s | 5.3 MB 00:00 2024-03-29T11:09:16.586 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:09:16.586 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2024-03-29T11:09:16.586 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2024-03-29T11:09:16.587 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2024-03-29T11:09:16.587 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2024-03-29T11:09:16.587 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2024-03-29T11:09:16.587 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /root 2024-03-29T11:09:16.587 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2024-03-29T11:09:16.587 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:16.618 INFO:teuthology.orchestra.run.smithi157.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 607 kB/s | 25 kB 00:00 2024-03-29T11:09:16.727 INFO:teuthology.orchestra.run.smithi157.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.7 MB/s | 742 kB 00:00 2024-03-29T11:09:16.794 INFO:teuthology.orchestra.run.smithi157.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T11:09:17.136 INFO:teuthology.orchestra.run.smithi157.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 15 MB/s | 5.1 MB 00:00 2024-03-29T11:09:17.178 INFO:teuthology.orchestra.run.smithi157.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-29T11:09:17.237 INFO:teuthology.orchestra.run.smithi157.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T11:09:17.287 INFO:teuthology.orchestra.run.smithi157.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.2 MB/s | 162 kB 00:00 2024-03-29T11:09:17.346 INFO:teuthology.orchestra.run.smithi157.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 5.4 MB/s | 320 kB 00:00 2024-03-29T11:09:17.404 INFO:teuthology.orchestra.run.smithi157.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T11:09:17.463 INFO:teuthology.orchestra.run.smithi157.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T11:09:17.515 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:09:17.539 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:09:17.539 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:09:17.540 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:09:17.540 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:09:17.540 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:09:17.540 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:17.541 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:09:17.571 INFO:teuthology.orchestra.run.smithi157.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-29T11:09:17.609 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:09:17.609 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:09:17.613 INFO:teuthology.orchestra.run.smithi157.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-29T11:09:17.647 INFO:teuthology.orchestra.run.smithi157.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:09:17.706 INFO:teuthology.orchestra.run.smithi157.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T11:09:17.739 INFO:teuthology.orchestra.run.smithi157.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2024-03-29T11:09:17.773 INFO:teuthology.orchestra.run.smithi157.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T11:09:17.806 INFO:teuthology.orchestra.run.smithi157.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.4 MB/s | 220 kB 00:00 2024-03-29T11:09:17.840 INFO:teuthology.orchestra.run.smithi157.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T11:09:17.873 INFO:teuthology.orchestra.run.smithi157.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T11:09:18.040 INFO:teuthology.orchestra.run.smithi157.stdout:(21/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 4.1 MB/s | 9.7 MB 00:02 2024-03-29T11:09:18.658 INFO:teuthology.orchestra.run.smithi157.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.6 MB/s | 20 MB 00:03 2024-03-29T11:09:18.683 INFO:teuthology.orchestra.run.smithi157.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 382 kB/s | 309 kB 00:00 2024-03-29T11:09:18.758 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:09:18.758 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:09:18.758 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:09:18.758 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:09:18.758 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:09:18.758 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:09:18.758 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:09:18.758 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:09:18.758 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:09:18.758 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:09:18.759 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:09:18.760 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:09:18.760 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:09:18.760 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:09:18.760 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:09:18.760 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:09:18.760 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:09:18.761 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:09:18.761 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:09:18.761 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:09:18.761 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:09:18.791 INFO:teuthology.orchestra.run.smithi157.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-29T11:09:18.917 INFO:teuthology.orchestra.run.smithi157.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2024-03-29T11:09:18.942 INFO:teuthology.orchestra.run.smithi157.stdout:(26/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 91 kB/s | 26 kB 00:00 2024-03-29T11:09:19.017 INFO:teuthology.orchestra.run.smithi157.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2024-03-29T11:09:19.242 INFO:teuthology.orchestra.run.smithi157.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 639 kB/s | 192 kB 00:00 2024-03-29T11:09:19.534 INFO:teuthology.orchestra.run.smithi157.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 12 MB/s | 3.4 MB 00:00 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.710 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.711 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:19.712 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:09:19.718 INFO:teuthology.orchestra.run.smithi157.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 17 MB/s | 3.1 MB 00:00 2024-03-29T11:09:20.018 INFO:teuthology.orchestra.run.smithi157.stdout:(31/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 761 kB/s | 838 kB 00:01 2024-03-29T11:09:20.027 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-test 2024-03-29T11:09:20.135 INFO:teuthology.orchestra.run.smithi157.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-03-29T11:09:20.138 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:20.139 INFO:teuthology.orchestra.run.smithi157.stdout:Total 12 MB/s | 58 MB 00:04 2024-03-29T11:09:20.139 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:09:20.177 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:09:20.177 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:09:20.595 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:09:20.651 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:09:20.651 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:09:20.745 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:09:20.746 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:09:20.746 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T11:09:20.746 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:09:20.746 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:09:20.746 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:09:20.746 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-29T11:09:20.746 INFO:teuthology.orchestra.run.smithi195.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:09:20.746 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:09:20.747 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:09:20.747 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:09:20.747 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:09:20.747 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:20.747 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:09:20.747 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:09:20.747 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-03-29T11:09:20.747 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:20.748 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 48 M 2024-03-29T11:09:20.748 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 204 M 2024-03-29T11:09:20.748 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:09:20.981 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:09:21.089 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 300 kB 00:00 2024-03-29T11:09:21.114 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 549 kB/s | 163 kB 00:00 2024-03-29T11:09:21.141 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T11:09:21.640 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 356 kB/s | 187 kB 00:00 2024-03-29T11:09:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:09:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:09:22.215 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:09:22.240 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:09:22.263 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:09:22.279 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:09:22.478 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:09:22.536 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:09:22.877 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:09:22.926 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:09:23.009 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:09:23.097 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:09:23.119 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:09:23.181 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:09:23.210 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:09:23.253 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:09:23.323 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:09:23.346 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:09:23.373 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-29T11:09:23.376 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:23.377 INFO:teuthology.orchestra.run.smithi195.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-29T11:09:23.392 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:09:23.401 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:09:23.413 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:09:23.414 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:09:23.436 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:09:23.546 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:09:23.579 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:09:23.620 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:09:23.661 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:09:23.689 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:09:23.723 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:09:23.798 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:09:23.799 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:09:24.030 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:09:24.061 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:09:24.134 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:09:24.165 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:09:24.210 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:09:24.386 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:09:24.465 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:09:24.513 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:09:24.553 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:09:24.594 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:09:24.618 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:09:24.667 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:09:24.696 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:09:24.759 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:09:24.827 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:09:24.857 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:09:27.046 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:09:27.095 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:09:27.645 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:09:27.682 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:09:27.763 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:09:27.764 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:09:27.764 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:09:27.764 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:27.800 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:09:29.108 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:09:29.192 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:09:31.521 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:09:31.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:09:31.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:09:31.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:09:31.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:09:31.522 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:09:31.896 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:09:31.896 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:31.896 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:09:31.897 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:31.897 INFO:teuthology.orchestra.run.smithi195.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:09:31.897 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:31.897 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:09:31.897 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:09:31.897 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:09:31.897 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:31.897 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:09:32.184 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph 2024-03-29T11:09:32.770 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:09:32.932 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:09:32.934 INFO:teuthology.orchestra.run.smithi195.stdout:========================================================================================= 2024-03-29T11:09:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T11:09:32.934 INFO:teuthology.orchestra.run.smithi195.stdout:========================================================================================= 2024-03-29T11:09:32.934 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:09:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:09:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:09:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:09:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:32.938 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:09:32.938 INFO:teuthology.orchestra.run.smithi195.stdout:========================================================================================= 2024-03-29T11:09:32.938 INFO:teuthology.orchestra.run.smithi195.stdout:Install 36 Packages 2024-03-29T11:09:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:32.939 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 30 M 2024-03-29T11:09:32.939 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 107 M 2024-03-29T11:09:32.939 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:09:33.562 INFO:teuthology.orchestra.run.smithi195.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-29T11:09:34.712 INFO:teuthology.orchestra.run.smithi195.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 1.5 MB 00:01 2024-03-29T11:09:34.862 INFO:teuthology.orchestra.run.smithi195.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 3.6 MB/s | 4.7 MB 00:01 2024-03-29T11:09:34.938 INFO:teuthology.orchestra.run.smithi195.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.2 MB/s | 247 kB 00:00 2024-03-29T11:09:35.066 INFO:teuthology.orchestra.run.smithi195.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T11:09:35.105 INFO:teuthology.orchestra.run.smithi195.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:09:35.138 INFO:teuthology.orchestra.run.smithi195.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2024-03-29T11:09:35.172 INFO:teuthology.orchestra.run.smithi195.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:09:35.206 INFO:teuthology.orchestra.run.smithi195.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:09:35.256 INFO:teuthology.orchestra.run.smithi195.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:09:35.331 INFO:teuthology.orchestra.run.smithi195.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:09:35.365 INFO:teuthology.orchestra.run.smithi195.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:09:35.398 INFO:teuthology.orchestra.run.smithi195.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:09:35.449 INFO:teuthology.orchestra.run.smithi195.stdout:(14/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86 971 kB/s | 2.1 MB 00:02 2024-03-29T11:09:35.474 INFO:teuthology.orchestra.run.smithi195.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-29T11:09:35.591 INFO:teuthology.orchestra.run.smithi195.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 881 kB/s | 125 kB 00:00 2024-03-29T11:09:35.633 INFO:teuthology.orchestra.run.smithi195.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 140 kB/s | 22 kB 00:00 2024-03-29T11:09:35.716 INFO:teuthology.orchestra.run.smithi195.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-03-29T11:09:35.742 INFO:teuthology.orchestra.run.smithi195.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 400 kB/s | 43 kB 00:00 2024-03-29T11:09:35.801 INFO:teuthology.orchestra.run.smithi195.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-29T11:09:35.842 INFO:teuthology.orchestra.run.smithi195.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 258 kB/s | 11 kB 00:00 2024-03-29T11:09:35.884 INFO:teuthology.orchestra.run.smithi195.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-03-29T11:09:35.910 INFO:teuthology.orchestra.run.smithi195.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-29T11:09:35.935 INFO:teuthology.orchestra.run.smithi195.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-03-29T11:09:35.960 INFO:teuthology.orchestra.run.smithi195.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-03-29T11:09:35.985 INFO:teuthology.orchestra.run.smithi195.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-03-29T11:09:36.011 INFO:teuthology.orchestra.run.smithi195.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 917 kB/s | 46 kB 00:00 2024-03-29T11:09:36.036 INFO:teuthology.orchestra.run.smithi195.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T11:09:36.095 INFO:teuthology.orchestra.run.smithi195.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-03-29T11:09:36.120 INFO:teuthology.orchestra.run.smithi195.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-29T11:09:36.145 INFO:teuthology.orchestra.run.smithi195.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-29T11:09:36.171 INFO:teuthology.orchestra.run.smithi195.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-03-29T11:09:36.229 INFO:teuthology.orchestra.run.smithi195.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-29T11:09:36.271 INFO:teuthology.orchestra.run.smithi195.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 478 kB/s | 20 kB 00:00 2024-03-29T11:09:36.296 INFO:teuthology.orchestra.run.smithi195.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-29T11:09:36.672 INFO:teuthology.orchestra.run.smithi195.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 8.6 MB/s | 17 MB 00:01 2024-03-29T11:09:36.675 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:36.676 INFO:teuthology.orchestra.run.smithi195.stdout:Total 7.9 MB/s | 30 MB 00:03 2024-03-29T11:09:36.698 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:09:36.739 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:09:36.739 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:09:37.150 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:09:37.151 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:09:37.529 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:09:37.529 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2024-03-29T11:09:37.529 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2024-03-29T11:09:37.529 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2024-03-29T11:09:37.530 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2024-03-29T11:09:37.530 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2024-03-29T11:09:37.530 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /root 2024-03-29T11:09:37.530 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2024-03-29T11:09:37.530 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:38.475 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:09:38.486 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:09:38.499 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:09:38.499 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:09:38.500 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:09:38.500 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:09:38.500 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:09:38.500 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:38.501 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:09:38.549 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:09:38.569 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:09:38.570 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:09:38.582 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:09:38.635 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:09:38.714 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:09:38.775 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:09:38.808 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:09:38.900 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:09:38.978 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:09:39.019 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:09:39.091 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:09:39.145 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:09:39.192 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:09:39.240 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:09:39.295 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:09:39.481 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:09:39.537 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:09:39.576 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:09:39.611 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:09:39.641 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:09:39.674 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:09:39.674 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:09:39.674 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:09:39.674 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:09:39.674 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:09:39.674 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:09:39.675 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:09:39.676 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:09:39.677 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:09:39.677 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:09:39.677 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:09:39.718 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:09:39.751 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:09:39.785 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:09:39.825 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:09:39.860 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:09:39.895 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:09:39.983 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:09:40.026 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:09:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:09:40.234 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:09:40.353 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:09:40.508 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:09:40.531 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:09:40.556 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:09:40.556 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:09:40.556 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:09:40.556 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:09:40.557 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:09:40.557 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:40.721 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:09:40.721 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:40.721 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2024-03-29T11:09:40.721 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.722 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:09:40.723 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:09:40.723 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:09:40.723 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.723 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:09:40.723 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.723 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:09:40.723 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:09:40.723 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:09:40.723 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:40.724 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:09:41.085 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-test 2024-03-29T11:09:41.636 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:09:41.794 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:09:41.795 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:09:41.795 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T11:09:41.795 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:09:41.795 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:09:41.795 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:09:41.795 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-29T11:09:41.795 INFO:teuthology.orchestra.run.smithi157.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:09:41.796 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:09:41.796 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:09:41.796 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:09:41.796 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:09:41.796 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:41.796 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:09:41.796 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:09:41.796 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2024-03-29T11:09:41.796 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:41.797 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 48 M 2024-03-29T11:09:41.797 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 204 M 2024-03-29T11:09:41.797 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:09:41.982 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:09:42.008 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:09:42.008 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:09:42.008 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:09:42.009 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:09:42.009 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:09:42.009 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:42.242 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:09:42.292 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T11:09:42.326 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:09:42.443 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 442 kB/s | 163 kB 00:00 2024-03-29T11:09:42.449 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:09:42.468 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T11:09:42.472 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:09:42.472 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:09:42.472 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:09:42.472 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:09:42.472 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:09:42.472 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:42.683 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:09:42.706 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:09:42.706 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:09:42.706 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:09:42.706 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:09:42.706 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:09:42.706 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:42.773 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:09:44.385 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-29T11:09:44.388 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:44.388 INFO:teuthology.orchestra.run.smithi157.stdout:Total 19 MB/s | 48 MB 00:02 2024-03-29T11:09:44.403 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:09:44.425 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:09:44.426 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:09:44.454 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:09:44.455 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:09:44.456 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:09:44.457 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:09:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:09:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:09:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:09:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:09:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:09:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:09:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:09:44.458 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:09:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:09:44.459 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:09:44.808 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:09:44.808 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:09:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:09:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:45.294 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:09:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:09:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:09:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:09:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:09:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:09:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:09:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:09:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:09:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:45.297 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:09:45.378 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:09:45.444 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:09:45.482 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:09:45.514 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:09:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:09:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:09:45.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base 2024-03-29T11:09:46.186 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:09:46.285 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:09:46.337 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:09:46.338 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-29T11:09:46.338 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:09:46.402 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install cephadm 2024-03-29T11:09:46.970 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:09:47.121 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:09:47.122 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:09:47.122 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T11:09:47.122 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:09:47.122 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:09:47.122 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:09:47.123 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:47.123 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:09:47.123 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:09:47.123 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-29T11:09:47.123 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:47.123 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 747 k 2024-03-29T11:09:47.123 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 754 k 2024-03-29T11:09:47.123 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:09:47.559 INFO:teuthology.orchestra.run.smithi195.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T11:09:47.559 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:47.560 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T11:09:47.560 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:09:47.565 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:09:47.565 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:09:47.571 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:09:47.571 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:09:47.641 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:09:48.075 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:09:48.165 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:09:48.574 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:09:48.853 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:09:48.853 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:48.853 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:09:48.853 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:09:48.853 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:48.853 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:09:49.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:09:49.630 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:09:49.783 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:09:49.784 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:09:49.784 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-29T11:09:49.784 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:09:49.784 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:09:49.784 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:09:49.784 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:49.784 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:09:49.784 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:09:49.784 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-29T11:09:49.784 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:49.785 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 145 k 2024-03-29T11:09:49.785 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 436 k 2024-03-29T11:09:49.785 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:09:50.080 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:09:50.132 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 420 kB/s | 145 kB 00:00 2024-03-29T11:09:50.132 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:50.133 INFO:teuthology.orchestra.run.smithi195.stdout:Total 418 kB/s | 145 kB 00:00 2024-03-29T11:09:50.133 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:09:50.140 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:09:50.140 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:09:50.154 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:09:50.193 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:09:50.193 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:09:50.330 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:09:50.399 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:09:50.424 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:09:50.424 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:09:50.424 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:09:50.424 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:09:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:51.448 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:09:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:51.448 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:51.449 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:09:51.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr 2024-03-29T11:09:52.267 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:09:52.365 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:09:52.418 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:09:52.419 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-29T11:09:52.419 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:09:52.484 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:09:52.617 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:09:52.617 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:09:52.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:09:52.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:09:52.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:09:52.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:09:52.974 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:09:52.974 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:52.974 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:09:52.974 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:52.974 INFO:teuthology.orchestra.run.smithi157.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:09:52.974 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:09:52.974 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:09:52.974 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:09:52.974 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:09:52.975 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:52.975 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:09:53.041 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:09:53.191 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:09:53.192 INFO:teuthology.orchestra.run.smithi195.stdout:====================================================================================== 2024-03-29T11:09:53.192 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout:====================================================================================== 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:09:53.193 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:09:53.194 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:09:53.194 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:53.194 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:09:53.194 INFO:teuthology.orchestra.run.smithi195.stdout:====================================================================================== 2024-03-29T11:09:53.194 INFO:teuthology.orchestra.run.smithi195.stdout:Install 12 Packages 2024-03-29T11:09:53.194 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:53.195 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 6.9 M 2024-03-29T11:09:53.195 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 26 M 2024-03-29T11:09:53.195 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:09:53.259 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph 2024-03-29T11:09:53.850 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:09:53.968 INFO:teuthology.orchestra.run.smithi195.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 73 kB/s | 15 kB 00:00 2024-03-29T11:09:54.014 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:09:54.016 INFO:teuthology.orchestra.run.smithi157.stdout:========================================================================================= 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout:========================================================================================= 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:09:54.017 INFO:teuthology.orchestra.run.smithi157.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:09:54.018 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:09:54.019 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:09:54.020 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:09:54.020 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:09:54.020 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:09:54.020 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:09:54.020 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:54.020 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:09:54.020 INFO:teuthology.orchestra.run.smithi157.stdout:========================================================================================= 2024-03-29T11:09:54.020 INFO:teuthology.orchestra.run.smithi157.stdout:Install 36 Packages 2024-03-29T11:09:54.020 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:09:54.021 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 30 M 2024-03-29T11:09:54.021 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 107 M 2024-03-29T11:09:54.021 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:09:54.119 INFO:teuthology.orchestra.run.smithi195.stdout:(2/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 5.0 MB/s | 1.8 MB 00:00 2024-03-29T11:09:54.143 INFO:teuthology.orchestra.run.smithi195.stdout:(3/12): ceph-grafana-dashboards-19.0.0-1439.g5d 82 kB/s | 31 kB 00:00 2024-03-29T11:09:54.402 INFO:teuthology.orchestra.run.smithi195.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-03-29T11:09:54.427 INFO:teuthology.orchestra.run.smithi195.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 772 kB/s | 238 kB 00:00 2024-03-29T11:09:54.494 INFO:teuthology.orchestra.run.smithi195.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-03-29T11:09:54.496 INFO:teuthology.orchestra.run.smithi157.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 42 kB/s | 6.6 kB 00:00 2024-03-29T11:09:54.520 INFO:teuthology.orchestra.run.smithi195.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.2 MB/s | 863 kB 00:00 2024-03-29T11:09:54.645 INFO:teuthology.orchestra.run.smithi195.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.2 MB/s | 548 kB 00:00 2024-03-29T11:09:54.670 INFO:teuthology.orchestra.run.smithi195.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 963 kB/s | 144 kB 00:00 2024-03-29T11:09:54.695 INFO:teuthology.orchestra.run.smithi195.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 609 kB/s | 31 kB 00:00 2024-03-29T11:09:54.738 INFO:teuthology.orchestra.run.smithi157.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.7 MB/s | 1.5 MB 00:00 2024-03-29T11:09:54.746 INFO:teuthology.orchestra.run.smithi195.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-03-29T11:09:54.946 INFO:teuthology.orchestra.run.smithi195.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.5 MB/s | 2.0 MB 00:00 2024-03-29T11:09:54.949 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:54.949 INFO:teuthology.orchestra.run.smithi195.stdout:Total 3.9 MB/s | 6.9 MB 00:01 2024-03-29T11:09:54.967 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:09:54.984 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:09:54.985 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:09:55.063 INFO:teuthology.orchestra.run.smithi157.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 8.3 MB/s | 4.7 MB 00:00 2024-03-29T11:09:55.139 INFO:teuthology.orchestra.run.smithi157.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.2 MB/s | 247 kB 00:00 2024-03-29T11:09:55.153 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:09:55.154 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:09:55.222 INFO:teuthology.orchestra.run.smithi157.stdout:(5/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.4 MB/s | 2.1 MB 00:00 2024-03-29T11:09:55.289 INFO:teuthology.orchestra.run.smithi157.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:09:55.314 INFO:teuthology.orchestra.run.smithi157.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2024-03-29T11:09:55.339 INFO:teuthology.orchestra.run.smithi157.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 631 kB/s | 32 kB 00:00 2024-03-29T11:09:55.365 INFO:teuthology.orchestra.run.smithi157.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-03-29T11:09:55.390 INFO:teuthology.orchestra.run.smithi157.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-29T11:09:55.415 INFO:teuthology.orchestra.run.smithi157.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:09:55.467 INFO:teuthology.orchestra.run.smithi157.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:09:55.491 INFO:teuthology.orchestra.run.smithi157.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T11:09:55.516 INFO:teuthology.orchestra.run.smithi157.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:09:55.541 INFO:teuthology.orchestra.run.smithi157.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:09:55.625 INFO:teuthology.orchestra.run.smithi157.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-29T11:09:55.661 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:09:55.738 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:09:55.781 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:09:55.792 INFO:teuthology.orchestra.run.smithi157.stdout:(17/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 16 MB/s | 17 MB 00:01 2024-03-29T11:09:55.809 INFO:teuthology.orchestra.run.smithi157.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 427 kB/s | 125 kB 00:00 2024-03-29T11:09:55.827 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:09:55.859 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:09:55.943 INFO:teuthology.orchestra.run.smithi157.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 289 kB/s | 43 kB 00:00 2024-03-29T11:09:55.965 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:09:55.968 INFO:teuthology.orchestra.run.smithi157.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 441 kB/s | 151 kB 00:00 2024-03-29T11:09:56.026 INFO:teuthology.orchestra.run.smithi157.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-29T11:09:56.052 INFO:teuthology.orchestra.run.smithi157.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 708 kB/s | 172 kB 00:00 2024-03-29T11:09:56.085 INFO:teuthology.orchestra.run.smithi157.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 301 kB/s | 18 kB 00:00 2024-03-29T11:09:56.090 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:09:56.111 INFO:teuthology.orchestra.run.smithi157.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-03-29T11:09:56.136 INFO:teuthology.orchestra.run.smithi157.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-29T11:09:56.161 INFO:teuthology.orchestra.run.smithi157.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-03-29T11:09:56.186 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:09:56.195 INFO:teuthology.orchestra.run.smithi157.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 234 kB/s | 20 kB 00:00 2024-03-29T11:09:56.220 INFO:teuthology.orchestra.run.smithi157.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 550 kB/s | 46 kB 00:00 2024-03-29T11:09:56.229 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:09:56.246 INFO:teuthology.orchestra.run.smithi157.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 938 kB/s | 79 kB 00:00 2024-03-29T11:09:56.267 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:09:56.279 INFO:teuthology.orchestra.run.smithi157.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-29T11:09:56.298 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:09:56.304 INFO:teuthology.orchestra.run.smithi157.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-29T11:09:56.329 INFO:teuthology.orchestra.run.smithi157.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-29T11:09:56.355 INFO:teuthology.orchestra.run.smithi157.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-03-29T11:09:56.372 INFO:teuthology.orchestra.run.smithi157.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-03-29T11:09:56.422 INFO:teuthology.orchestra.run.smithi157.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-03-29T11:09:56.447 INFO:teuthology.orchestra.run.smithi157.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-03-29T11:09:56.450 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:56.450 INFO:teuthology.orchestra.run.smithi157.stdout:Total 12 MB/s | 30 MB 00:02 2024-03-29T11:09:56.475 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:09:56.518 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:09:56.518 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:09:56.538 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:09:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:09:56.926 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:09:57.053 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:09:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:09:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:09:57.399 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:09:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:09:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:09:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:09:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:09:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:09:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:09:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:09:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:09:57.400 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:09:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:09:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:09:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:09:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:57.739 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:09:57.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:09:58.268 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:09:58.325 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:09:58.353 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:09:58.394 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:09:58.465 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:09:58.518 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:09:58.525 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:09:58.550 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:09:58.643 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:09:58.676 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:09:58.677 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-03-29T11:09:58.677 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T11:09:58.677 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-03-29T11:09:58.677 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:09:58.677 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:09:58.678 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:09:58.679 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:09:58.679 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:58.679 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:09:58.679 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-03-29T11:09:58.679 INFO:teuthology.orchestra.run.smithi195.stdout:Install 12 Packages 2024-03-29T11:09:58.679 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:09:58.680 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 37 M 2024-03-29T11:09:58.680 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 209 M 2024-03-29T11:09:58.680 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:09:58.713 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:09:58.762 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:09:58.834 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:09:58.887 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:09:58.926 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:09:58.975 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:09:59.029 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:09:59.215 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:09:59.263 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:09:59.301 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:09:59.335 INFO:teuthology.orchestra.run.smithi195.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 271 kB/s | 32 kB 00:00 2024-03-29T11:09:59.344 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:09:59.374 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:09:59.394 INFO:teuthology.orchestra.run.smithi195.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 487 kB/s | 17 kB 00:00 2024-03-29T11:09:59.438 INFO:teuthology.orchestra.run.smithi195.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 821 kB/s | 34 kB 00:00 2024-03-29T11:09:59.459 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:09:59.493 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:09:59.526 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:09:59.567 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:09:59.602 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:09:59.603 INFO:teuthology.orchestra.run.smithi195.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-29T11:09:59.645 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:09:59.735 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:09:59.761 INFO:teuthology.orchestra.run.smithi195.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 31 MB/s | 4.8 MB 00:00 2024-03-29T11:09:59.800 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:09:59.820 INFO:teuthology.orchestra.run.smithi195.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2024-03-29T11:09:59.846 INFO:teuthology.orchestra.run.smithi195.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.5 MB/s | 217 kB 00:00 2024-03-29T11:09:59.874 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:09:59.946 INFO:teuthology.orchestra.run.smithi195.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 8.0 MB/s | 803 kB 00:00 2024-03-29T11:10:00.010 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:10:00.021 INFO:teuthology.orchestra.run.smithi195.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-03-29T11:10:00.137 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:10:00.213 INFO:teuthology.orchestra.run.smithi195.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-29T11:10:00.288 INFO:teuthology.orchestra.run.smithi195.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 927 kB/s | 245 kB 00:00 2024-03-29T11:10:00.300 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:10:00.330 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:10:00.369 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:10:00.369 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:00.369 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:10:00.369 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:10:00.369 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:10:00.369 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:00.638 INFO:teuthology.orchestra.run.smithi195.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.2 MB/s | 7.4 MB 00:01 2024-03-29T11:10:00.641 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:00.641 INFO:teuthology.orchestra.run.smithi195.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-29T11:10:00.682 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:10:00.706 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:10:00.706 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:10:01.077 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:10:01.077 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:10:01.807 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:10:01.814 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:10:01.843 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:10:01.844 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:01.844 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:10:01.844 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:10:01.844 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:10:01.844 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:01.879 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:10:01.912 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:10:02.005 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:10:02.055 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:10:02.228 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:10:02.253 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:10:02.282 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:10:02.305 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:10:02.305 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:02.305 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:10:02.305 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:10:02.306 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:10:02.306 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:02.334 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:10:02.516 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:10:02.540 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:10:02.540 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:02.540 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:10:02.540 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:10:02.540 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:10:02.540 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:02.605 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:10:02.983 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:10:03.092 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:10:04.349 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:10:04.440 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:10:04.441 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:10:04.442 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:10:04.443 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:10:05.378 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:10:05.403 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:10:05.407 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:10:05.407 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:10:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:10:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:10:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:10:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:10:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:10:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:10:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:10:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:10:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:10:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:10:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:05.410 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:05.769 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base 2024-03-29T11:10:06.346 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:06.446 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:06.499 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:06.500 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-29T11:10:06.500 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:06.576 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install cephadm 2024-03-29T11:10:07.142 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:07.292 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:07.293 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:07.293 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T11:10:07.293 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:07.293 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:07.293 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:10:07.293 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:07.294 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:07.294 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:07.294 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-29T11:10:07.294 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:07.294 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 747 k 2024-03-29T11:10:07.294 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 754 k 2024-03-29T11:10:07.294 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:07.581 INFO:teuthology.orchestra.run.smithi157.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.5 MB/s | 747 kB 00:00 2024-03-29T11:10:07.581 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:07.582 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.5 MB/s | 747 kB 00:00 2024-03-29T11:10:07.582 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:07.587 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:07.587 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:07.592 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:07.593 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:07.667 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:07.908 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:10:07.908 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:10:07.908 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:10:07.908 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:10:07.909 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:10:07.909 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:10:07.909 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:10:07.909 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:10:07.909 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:10:07.909 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:10:07.909 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:10:07.909 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:10:08.155 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:08.227 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:08.411 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:10:08.411 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:08.411 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:10:08.411 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:08.411 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:10:08.411 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:10:08.411 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:10:08.411 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:10:08.411 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:10:08.412 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:10:08.412 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:10:08.412 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:10:08.412 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:10:08.412 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:10:08.412 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:10:08.412 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:08.412 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:08.610 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:08.755 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:10:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:08.887 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:08.887 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:09.119 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:10:09.325 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:09.477 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:09.478 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-29T11:10:09.478 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T11:10:09.478 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-29T11:10:09.478 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:10:09.478 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:10:09.478 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:09.479 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:10:09.480 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-29T11:10:09.480 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2024-03-29T11:10:09.480 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:09.480 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 2.0 M 2024-03-29T11:10:09.480 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 26 M 2024-03-29T11:10:09.480 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:10:09.695 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:09.844 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:09.845 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-29T11:10:09.845 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:09.845 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:10:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:09.845 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:09.845 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:09.846 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-29T11:10:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:09.846 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 145 k 2024-03-29T11:10:09.846 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 436 k 2024-03-29T11:10:09.846 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:09.929 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 18 kB 00:00 2024-03-29T11:10:09.946 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 140 kB/s | 24 kB 00:00 2024-03-29T11:10:09.963 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 273 kB/s | 52 kB 00:00 2024-03-29T11:10:10.063 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 174 kB 00:00 2024-03-29T11:10:10.070 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 651 kB/s | 145 kB 00:00 2024-03-29T11:10:10.070 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:10.070 INFO:teuthology.orchestra.run.smithi157.stdout:Total 645 kB/s | 145 kB 00:00 2024-03-29T11:10:10.071 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:10.078 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:10.078 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:10.132 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:10.132 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:10.172 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 586 kB/s | 132 kB 00:00 2024-03-29T11:10:10.197 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 901 kB/s | 211 kB 00:00 2024-03-29T11:10:10.214 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 284 kB/s | 43 kB 00:00 2024-03-29T11:10:10.256 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 385 kB/s | 32 kB 00:00 2024-03-29T11:10:10.272 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-03-29T11:10:10.273 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:10.340 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-03-29T11:10:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:10.365 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.3 MB/s | 203 kB 00:00 2024-03-29T11:10:10.370 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:10.370 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:10.370 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:10:10.370 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:10.406 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-29T11:10:10.457 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.1 MB/s | 1.0 MB 00:00 2024-03-29T11:10:10.459 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:10.459 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-29T11:10:10.459 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:10:10.472 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:10:10.473 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:10:10.563 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:10:10.563 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:10:11.075 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:10:11.254 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:10:11.307 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:10:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:11.312 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:11.312 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:11.363 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:10:11.400 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:10:11.441 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:10:11.544 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:10:11.564 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr 2024-03-29T11:10:11.652 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:10:11.701 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:10:12.138 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:12.240 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:12.292 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:12.293 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-29T11:10:12.293 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:12.374 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:10:12.419 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:10:12.474 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:10:12.507 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:10:12.555 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:10:12.577 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:10:12.937 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:13.089 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:13.090 INFO:teuthology.orchestra.run.smithi157.stdout:====================================================================================== 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout:====================================================================================== 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:10:13.091 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:10:13.092 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:10:13.092 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:13.092 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:13.092 INFO:teuthology.orchestra.run.smithi157.stdout:====================================================================================== 2024-03-29T11:10:13.092 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-03-29T11:10:13.092 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:13.092 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 6.9 M 2024-03-29T11:10:13.092 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2024-03-29T11:10:13.092 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:13.326 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:10:13.326 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:10:13.326 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:10:13.327 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:10:13.327 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:10:13.327 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:10:13.327 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:10:13.327 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:10:13.327 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:10:13.327 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:10:13.327 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:10:13.327 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:10:13.327 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:10:13.673 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 83 kB/s | 15 kB 00:00 2024-03-29T11:10:13.683 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:10:13.683 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:13.683 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:10:13.683 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:13.683 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:10:13.683 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:10:13.683 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:10:13.683 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:10:13.683 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:10:13.683 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:10:13.684 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:10:13.684 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:10:13.684 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:10:13.684 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:10:13.684 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:10:13.684 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:10:13.684 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:13.684 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:13.732 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 129 kB/s | 31 kB 00:00 2024-03-29T11:10:13.807 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.1 MB/s | 238 kB 00:00 2024-03-29T11:10:13.849 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.7 MB/s | 1.0 MB 00:00 2024-03-29T11:10:13.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:10:13.999 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-03-29T11:10:14.125 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.8 MB/s | 1.8 MB 00:00 2024-03-29T11:10:14.242 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 81 kB/s | 19 kB 00:00 2024-03-29T11:10:14.342 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:10:14.410 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 462 kB/s | 31 kB 00:00 2024-03-29T11:10:14.434 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 756 kB/s | 144 kB 00:00 2024-03-29T11:10:14.493 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T11:10:14.495 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:14.649 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:14.649 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:10:14.650 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2024-03-29T11:10:14.651 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-03-29T11:10:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:14.651 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 6.8 M 2024-03-29T11:10:14.651 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 33 M 2024-03-29T11:10:14.651 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:10:14.676 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.7 MB/s | 2.0 MB 00:00 2024-03-29T11:10:14.681 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:14.681 INFO:teuthology.orchestra.run.smithi157.stdout:Total 4.3 MB/s | 6.9 MB 00:01 2024-03-29T11:10:14.699 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:14.716 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:14.716 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:14.911 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:14.911 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:14.958 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 630 kB/s | 138 kB 00:00 2024-03-29T11:10:15.033 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T11:10:15.250 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 301 kB/s | 58 kB 00:00 2024-03-29T11:10:15.292 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T11:10:15.317 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 766 kB/s | 51 kB 00:00 2024-03-29T11:10:15.417 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:15.493 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:10:15.528 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:10:15.575 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:10:15.614 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:10:15.717 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:10:15.726 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 251 kB/s | 248 kB 00:00 2024-03-29T11:10:15.729 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:15.729 INFO:teuthology.orchestra.run.smithi195.stdout:Total 6.3 MB/s | 6.8 MB 00:01 2024-03-29T11:10:15.742 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:10:15.752 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:10:15.752 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:10:15.840 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:10:15.841 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:10:15.842 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:10:15.934 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:10:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:10:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:10:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:10:16.129 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:10:16.295 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:10:16.311 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:10:16.329 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:10:16.352 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:10:16.868 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:10:16.948 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:10:17.008 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:10:17.032 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:10:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:10:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:10:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:10:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:10:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:10:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:10:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:10:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:10:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:10:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:10:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:10:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:10:17.471 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:10:17.472 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:17.472 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:17.472 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:10:17.472 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:17.472 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:17.472 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:17.472 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:10:17.472 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:10:17.473 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:10:17.473 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:10:17.473 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:10:17.473 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:10:17.473 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:10:17.473 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:10:17.473 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:17.473 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:17.759 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:10:17.896 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:10:17.897 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:10:17.897 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:10:17.897 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:10:17.897 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:10:17.897 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:10:18.184 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:10:18.184 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:18.184 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:10:18.184 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:18.184 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:10:18.184 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:10:18.184 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:10:18.184 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:10:18.184 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:10:18.184 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:18.185 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:18.327 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:18.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-fuse 2024-03-29T11:10:18.479 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:18.480 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:10:18.481 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:10:18.482 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:10:18.482 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:10:18.482 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:10:18.482 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:18.482 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:18.482 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-29T11:10:18.482 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-03-29T11:10:18.482 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:18.483 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 37 M 2024-03-29T11:10:18.483 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 209 M 2024-03-29T11:10:18.483 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:18.978 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:19.128 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:19.130 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:19.130 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T11:10:19.130 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:19.130 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 964 k 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 2.8 M 2024-03-29T11:10:19.131 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:10:19.166 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 242 kB/s | 32 kB 00:00 2024-03-29T11:10:19.199 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-29T11:10:19.269 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 565 kB/s | 79 kB 00:00 2024-03-29T11:10:19.274 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-29T11:10:19.300 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 348 kB/s | 34 kB 00:00 2024-03-29T11:10:19.500 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-29T11:10:19.567 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T11:10:19.592 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T11:10:19.768 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-29T11:10:19.810 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 195 kB 00:00 2024-03-29T11:10:19.836 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 885 kB 00:00 2024-03-29T11:10:19.837 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:19.838 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.3 MB/s | 964 kB 00:00 2024-03-29T11:10:19.838 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:10:19.858 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:10:19.858 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:10:19.931 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:10:19.932 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:10:19.968 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-29T11:10:19.993 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-29T11:10:20.085 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 7.0 MB/s | 7.4 MB 00:01 2024-03-29T11:10:20.088 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:20.088 INFO:teuthology.orchestra.run.smithi157.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-29T11:10:20.120 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:10:20.129 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:20.154 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:20.154 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:20.253 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:10:20.319 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:10:20.513 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:20.514 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:10:21.018 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:10:21.235 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:21.308 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:10:21.319 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:10:21.319 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:21.319 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:10:21.319 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:10:21.319 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:21.319 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:21.347 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:10:21.442 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:10:21.476 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:10:21.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-volume 2024-03-29T11:10:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:10:21.692 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:10:21.776 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:10:22.148 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:22.302 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:22.303 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-03-29T11:10:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:10:22.303 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:10:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-29T11:10:22.303 INFO:teuthology.orchestra.run.smithi195.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:10:22.303 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:10:22.303 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:22.304 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-03-29T11:10:22.304 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:22.304 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 332 k 2024-03-29T11:10:22.304 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 1.5 M 2024-03-29T11:10:22.304 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:10:22.438 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:10:22.524 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.2 MB/s | 264 kB 00:00 2024-03-29T11:10:22.555 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:10:22.582 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 247 kB/s | 69 kB 00:00 2024-03-29T11:10:22.583 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:22.584 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T11:10:22.584 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:10:22.592 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:10:22.592 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:10:22.628 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:10:22.629 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:10:22.779 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:10:22.965 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:10:23.010 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:10:23.026 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:10:23.026 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:23.026 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:10:23.026 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:23.788 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:10:23.796 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:10:24.110 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:10:24.110 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:24.110 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:10:24.110 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:24.110 INFO:teuthology.orchestra.run.smithi195.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:10:24.111 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:24.111 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:24.451 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados-devel 2024-03-29T11:10:24.848 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:10:24.874 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:10:25.035 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:25.185 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:25.185 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:25.185 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-29T11:10:25.186 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:25.186 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:10:25.186 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:10:25.186 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:25.186 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:10:25.186 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:25.186 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-29T11:10:25.186 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:25.187 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 127 k 2024-03-29T11:10:25.187 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 456 k 2024-03-29T11:10:25.187 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:10:25.798 INFO:teuthology.orchestra.run.smithi195.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 207 kB/s | 127 kB 00:00 2024-03-29T11:10:25.799 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:25.799 INFO:teuthology.orchestra.run.smithi195.stdout:Total 207 kB/s | 127 kB 00:00 2024-03-29T11:10:25.799 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:10:25.804 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:10:25.805 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:10:25.860 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:10:25.860 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:10:25.999 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:10:26.114 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:26.593 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:26.892 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:26.893 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:26.893 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:10:26.893 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:26.893 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:26.893 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:27.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs2 2024-03-29T11:10:27.407 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:10:27.408 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:10:27.408 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:10:27.408 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:10:27.408 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:10:27.408 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:10:27.408 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:10:27.408 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:10:27.409 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:10:27.409 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:10:27.409 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:10:27.409 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:10:27.673 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:27.774 INFO:teuthology.orchestra.run.smithi195.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:27.829 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:27.830 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-29T11:10:27.830 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:27.855 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:10:27.855 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:27.855 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:27.855 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:27.855 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:10:27.855 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:10:27.855 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:10:27.855 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:10:27.855 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:10:27.855 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:10:27.856 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:10:27.856 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:10:27.856 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:10:27.856 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:10:27.856 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:10:27.856 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:27.856 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:27.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs-devel 2024-03-29T11:10:28.264 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:10:28.463 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:28.616 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:28.617 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:28.617 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-29T11:10:28.617 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:28.617 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:10:28.617 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:10:28.617 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:28.617 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:10:28.617 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:28.618 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-29T11:10:28.618 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:28.618 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 31 k 2024-03-29T11:10:28.618 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 140 k 2024-03-29T11:10:28.618 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:10:28.793 INFO:teuthology.orchestra.run.smithi195.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 177 kB/s | 31 kB 00:00 2024-03-29T11:10:28.794 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:28.794 INFO:teuthology.orchestra.run.smithi195.stdout:Total 176 kB/s | 31 kB 00:00 2024-03-29T11:10:28.794 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:10:28.797 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:10:28.798 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:10:28.830 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:10:28.830 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:10:28.840 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:16 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:28.958 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:10:28.993 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:28.994 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-29T11:10:28.994 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T11:10:28.994 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:10:28.995 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:10:28.996 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:10:28.996 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:10:28.996 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:28.996 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:28.996 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-29T11:10:28.996 INFO:teuthology.orchestra.run.smithi157.stdout:Install 13 Packages 2024-03-29T11:10:28.996 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:28.997 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.0 M 2024-03-29T11:10:28.997 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2024-03-29T11:10:28.997 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:29.108 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:29.474 INFO:teuthology.orchestra.run.smithi157.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 24 kB 00:00 2024-03-29T11:10:29.495 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:29.499 INFO:teuthology.orchestra.run.smithi157.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2024-03-29T11:10:29.558 INFO:teuthology.orchestra.run.smithi157.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-29T11:10:29.583 INFO:teuthology.orchestra.run.smithi157.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-29T11:10:29.608 INFO:teuthology.orchestra.run.smithi157.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:10:29.625 INFO:teuthology.orchestra.run.smithi157.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T11:10:29.684 INFO:teuthology.orchestra.run.smithi157.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 158 kB/s | 52 kB 00:00 2024-03-29T11:10:29.734 INFO:teuthology.orchestra.run.smithi157.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 138 kB/s | 15 kB 00:00 2024-03-29T11:10:29.754 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:29.754 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:29.754 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:10:29.754 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:29.754 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:29.755 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:29.768 INFO:teuthology.orchestra.run.smithi157.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-03-29T11:10:29.860 INFO:teuthology.orchestra.run.smithi157.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 645 kB/s | 59 kB 00:00 2024-03-29T11:10:29.916 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados2 2024-03-29T11:10:29.952 INFO:teuthology.orchestra.run.smithi157.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 979 kB/s | 90 kB 00:00 2024-03-29T11:10:29.977 INFO:teuthology.orchestra.run.smithi157.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 692 kB/s | 203 kB 00:00 2024-03-29T11:10:30.102 INFO:teuthology.orchestra.run.smithi157.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-03-29T11:10:30.104 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:30.104 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-29T11:10:30.104 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:30.118 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:30.118 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:30.210 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:30.211 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:30.474 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:30.572 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:30.623 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:30.624 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-29T11:10:30.624 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:30.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librbd1 2024-03-29T11:10:30.726 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:30.897 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:10:30.953 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:10:31.009 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:10:31.046 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:10:31.087 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:10:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:10:31.243 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:31.281 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:10:31.341 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:31.363 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:10:31.392 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:31.393 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-29T11:10:31.393 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:31.446 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rados 2024-03-29T11:10:32.014 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:32.085 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:10:32.113 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:32.127 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:10:32.161 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:10:32.165 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:32.166 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-29T11:10:32.166 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:32.209 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:10:32.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rgw 2024-03-29T11:10:32.231 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:10:32.788 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:32.887 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:32.938 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:32.939 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-29T11:10:32.939 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:32.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-cephfs 2024-03-29T11:10:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:10:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:10:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:10:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:10:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:10:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:10:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:10:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:10:32.998 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:10:32.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:10:32.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:10:32.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:10:32.999 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:10:33.359 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:10:33.359 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:33.359 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:33.359 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:33.359 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:10:33.359 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:10:33.359 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:33.360 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:33.563 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:33.614 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:10:33.662 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:33.714 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:33.715 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-29T11:10:33.715 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:33.765 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rbd 2024-03-29T11:10:34.187 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:34.329 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:34.339 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:34.339 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:34.340 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-29T11:10:34.341 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2024-03-29T11:10:34.341 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:34.341 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 6.8 M 2024-03-29T11:10:34.341 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 33 M 2024-03-29T11:10:34.341 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:34.428 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:34.480 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:34.481 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-03-29T11:10:34.481 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:34.532 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-fuse 2024-03-29T11:10:34.729 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-29T11:10:34.879 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-29T11:10:34.979 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 332 kB/s | 138 kB 00:00 2024-03-29T11:10:35.054 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T11:10:35.080 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 290 kB/s | 58 kB 00:00 2024-03-29T11:10:35.099 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:35.105 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 407 kB/s | 51 kB 00:00 2024-03-29T11:10:35.107 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:35.107 INFO:teuthology.orchestra.run.smithi157.stdout:Total 8.9 MB/s | 6.8 MB 00:00 2024-03-29T11:10:35.121 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:35.131 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:35.131 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:35.221 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:35.221 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:35.246 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:35.247 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:35.247 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-03-29T11:10:35.247 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:35.248 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:10:35.248 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:10:35.248 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:35.248 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:10:35.248 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:35.248 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-29T11:10:35.248 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:35.248 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 85 k 2024-03-29T11:10:35.248 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 231 k 2024-03-29T11:10:35.248 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:10:35.514 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:35.522 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 312 kB/s | 85 kB 00:00 2024-03-29T11:10:35.523 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:35.523 INFO:teuthology.orchestra.run.smithi195.stdout:Total 310 kB/s | 85 kB 00:00 2024-03-29T11:10:35.523 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:10:35.529 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:10:35.529 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:10:35.585 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:10:35.586 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:10:35.682 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:10:35.722 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:10:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:10:35.837 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:36.191 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:36.258 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:10:36.334 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:10:36.395 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:10:36.418 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:10:36.496 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:36.497 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:36.497 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:10:36.497 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:36.497 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:36.497 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:36.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror 2024-03-29T11:10:37.245 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:10:37.245 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:10:37.245 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:10:37.245 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:10:37.245 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:10:37.245 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:10:37.323 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:37.482 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:37.483 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:37.483 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-03-29T11:10:37.483 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:37.483 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:10:37.483 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:10:37.484 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:37.484 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:10:37.484 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:37.484 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-29T11:10:37.484 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:37.484 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.1 M 2024-03-29T11:10:37.484 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 13 M 2024-03-29T11:10:37.484 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:10:37.551 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:10:37.551 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:37.551 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:37.551 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:37.552 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:10:37.552 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:10:37.552 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:10:37.552 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:10:37.552 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:10:37.552 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:37.552 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:37.824 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-fuse 2024-03-29T11:10:38.117 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 4.9 MB/s | 3.1 MB 00:00 2024-03-29T11:10:38.118 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:38.118 INFO:teuthology.orchestra.run.smithi195.stdout:Total 4.9 MB/s | 3.1 MB 00:00 2024-03-29T11:10:38.118 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:10:38.127 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:10:38.128 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:10:38.201 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:10:38.201 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:10:38.391 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:38.543 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:38.544 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2024-03-29T11:10:38.545 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:38.545 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 964 k 2024-03-29T11:10:38.545 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 2.8 M 2024-03-29T11:10:38.545 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:38.629 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:10:38.674 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 608 kB/s | 79 kB 00:00 2024-03-29T11:10:38.693 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:38.718 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:38.718 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:38.718 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:10:38.719 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:10:38.719 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:10:38.719 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:39.074 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 885 kB 00:00 2024-03-29T11:10:39.076 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:39.076 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.8 MB/s | 964 kB 00:00 2024-03-29T11:10:39.076 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:39.097 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:39.097 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:39.170 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:39.171 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:39.354 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:39.492 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:10:39.566 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:10:39.829 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:39.830 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:39.830 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:10:39.830 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:39.830 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:39.830 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:40.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-nbd 2024-03-29T11:10:40.262 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:10:40.263 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:10:40.567 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:10:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:40.568 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:10:40.568 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:40.568 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:40.661 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:08:52 AM UTC. 2024-03-29T11:10:40.810 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-29T11:10:40.811 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:40.811 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-03-29T11:10:40.811 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:40.811 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-29T11:10:40.811 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:10:40.811 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:40.811 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-29T11:10:40.811 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-29T11:10:40.811 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-29T11:10:40.811 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:40.812 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 171 k 2024-03-29T11:10:40.812 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 494 k 2024-03-29T11:10:40.812 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-29T11:10:40.880 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-volume 2024-03-29T11:10:41.037 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 762 kB/s | 171 kB 00:00 2024-03-29T11:10:41.038 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:41.038 INFO:teuthology.orchestra.run.smithi195.stdout:Total 756 kB/s | 171 kB 00:00 2024-03-29T11:10:41.038 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-29T11:10:41.046 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-29T11:10:41.046 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-29T11:10:41.102 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-29T11:10:41.102 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-29T11:10:41.246 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-29T11:10:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:41.467 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:41.618 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:41.619 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:41.619 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 332 k 2024-03-29T11:10:41.620 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.5 M 2024-03-29T11:10:41.621 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:41.754 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 516 kB/s | 69 kB 00:00 2024-03-29T11:10:41.786 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:42.004 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 687 kB/s | 264 kB 00:00 2024-03-29T11:10:42.005 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:42.006 INFO:teuthology.orchestra.run.smithi157.stdout:Total 861 kB/s | 332 kB 00:00 2024-03-29T11:10:42.006 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:42.014 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:42.014 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:42.054 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-29T11:10:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:10:42.054 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-29T11:10:42.055 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:42.055 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:42.259 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:42.394 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:10:42.449 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:10:42.465 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:10:42.465 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:42.465 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:10:42.465 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:43.214 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:10:43.565 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:10:43.565 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:43.565 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:43.566 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:43.566 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:10:43.566 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:43.566 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:43.898 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados-devel 2024-03-29T11:10:44.478 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:44.629 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:44.631 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:44.631 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-29T11:10:44.631 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:44.631 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:44.631 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:10:44.631 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:44.631 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:44.631 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:44.632 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-29T11:10:44.632 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:44.632 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 127 k 2024-03-29T11:10:44.632 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 456 k 2024-03-29T11:10:44.632 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:44.826 INFO:teuthology.orchestra.run.smithi157.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 649 kB/s | 127 kB 00:00 2024-03-29T11:10:44.827 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:44.827 INFO:teuthology.orchestra.run.smithi157.stdout:Total 645 kB/s | 127 kB 00:00 2024-03-29T11:10:44.827 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:44.832 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:44.832 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:44.892 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:44.892 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:45.591 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:45.865 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:45.865 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:45.865 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:45.865 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:45.865 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:45.866 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:46.102 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs2 2024-03-29T11:10:46.679 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:46.781 INFO:teuthology.orchestra.run.smithi157.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:46.837 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:46.838 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-29T11:10:46.838 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:46.920 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs-devel 2024-03-29T11:10:47.494 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:47.647 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:47.647 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 31 k 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 140 k 2024-03-29T11:10:47.648 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:47.821 INFO:teuthology.orchestra.run.smithi157.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 181 kB/s | 31 kB 00:00 2024-03-29T11:10:47.822 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:47.822 INFO:teuthology.orchestra.run.smithi157.stdout:Total 179 kB/s | 31 kB 00:00 2024-03-29T11:10:47.822 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:47.825 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:47.825 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:47.855 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:47.855 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:48.009 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:48.121 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:48.503 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:48.773 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:48.773 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:48.773 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:48.773 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:48.774 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:48.774 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:48.943 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados2 2024-03-29T11:10:49.512 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:49.610 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:49.661 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:49.662 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-29T11:10:49.662 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:49.710 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librbd1 2024-03-29T11:10:50.274 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:50.370 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:50.421 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:50.422 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-29T11:10:50.422 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:50.476 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rados 2024-03-29T11:10:51.052 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:51.151 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:51.204 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:51.205 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-29T11:10:51.205 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:51.253 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rgw 2024-03-29T11:10:51.822 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:51.920 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:51.971 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:51.972 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-29T11:10:51.972 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:52.031 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-cephfs 2024-03-29T11:10:52.603 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:52.701 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:52.754 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:52.754 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-29T11:10:52.755 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:52.798 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rbd 2024-03-29T11:10:53.369 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:53.468 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:53.522 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:53.522 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-29T11:10:53.522 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:53.576 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-fuse 2024-03-29T11:10:54.142 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:54.292 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:54.293 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:54.293 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-29T11:10:54.293 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:54.294 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:54.294 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:10:54.294 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:54.294 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:54.294 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:54.294 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-29T11:10:54.294 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:54.294 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 85 k 2024-03-29T11:10:54.294 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 231 k 2024-03-29T11:10:54.294 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:54.615 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 266 kB/s | 85 kB 00:00 2024-03-29T11:10:54.616 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:54.616 INFO:teuthology.orchestra.run.smithi157.stdout:Total 265 kB/s | 85 kB 00:00 2024-03-29T11:10:54.616 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:54.623 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:54.623 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:54.680 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:54.681 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:54.923 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:55.314 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:55.600 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:55.600 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:55.600 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:55.600 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:55.600 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:55.600 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:55.843 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-mirror 2024-03-29T11:10:56.443 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:56.604 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:56.605 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:56.605 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-29T11:10:56.605 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:56.605 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:56.605 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:10:56.605 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:56.605 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:56.606 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:56.606 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-29T11:10:56.606 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:56.606 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 3.1 M 2024-03-29T11:10:56.606 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 13 M 2024-03-29T11:10:56.606 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:10:57.002 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 7.8 MB/s | 3.1 MB 00:00 2024-03-29T11:10:57.003 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:57.003 INFO:teuthology.orchestra.run.smithi157.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2024-03-29T11:10:57.003 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:10:57.012 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:10:57.012 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:10:57.085 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:10:57.086 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:10:57.514 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:10:57.586 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:57.611 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:57.611 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:57.612 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:10:57.612 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:10:57.612 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:10:57.612 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:58.715 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:10:58.715 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:58.715 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:10:58.716 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:58.716 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:58.716 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:10:58.965 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-nbd 2024-03-29T11:10:59.544 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:09:12 AM UTC. 2024-03-29T11:10:59.695 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:10:59.696 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 171 k 2024-03-29T11:10:59.697 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 494 k 2024-03-29T11:10:59.697 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-29T11:11:00.013 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 542 kB/s | 171 kB 00:00 2024-03-29T11:11:00.014 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:00.014 INFO:teuthology.orchestra.run.smithi157.stdout:Total 540 kB/s | 171 kB 00:00 2024-03-29T11:11:00.014 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-29T11:11:00.022 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-29T11:11:00.022 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-29T11:11:00.078 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-29T11:11:00.079 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-29T11:11:00.222 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-29T11:11:00.337 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:00.756 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:01.023 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:01.023 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:01.023 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-29T11:11:01.024 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:01.024 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:01.024 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-29T11:11:01.271 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:01.272 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:11:01.419 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:11:01.440 INFO:teuthology.orchestra.run.smithi157.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:11:01.440 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:11:01.440 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:11:01.442 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:11:01.678 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:11:01.703 INFO:teuthology.orchestra.run.smithi195.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:11:01.704 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:11:01.704 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:11:01.706 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:11:01.706 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:01.706 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:11:01.734 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:01.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:11:01.775 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:11:01.775 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:01.776 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:11:01.803 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:11:01.870 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:01.870 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:11:01.901 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:11:01.971 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:11:01.971 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:01.971 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:11:01.998 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:11:02.068 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:02.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:11:02.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:11:02.166 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:11:02.166 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:02.167 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:11:02.193 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:11:02.261 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:02.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:11:02.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:11:02.359 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:11:02.455 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:11:02.455 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2024-03-29T11:11:02.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /var/log/ceph 2024-03-29T11:11:02.488 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:11:02.488 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:11:02.525 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:11:02.554 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:11:02.554 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:11:02.593 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:11:02.631 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:11:02.632 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:11:02.632 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622676', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEzpVDeER3LSqzcPCX2onA/tmlitjWMAGbeI4lRfZ3+IlWdRVJ1SVubTqvC3OzjxhI6SW7/YDv4bE6CdsPrdO1s=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKZbT9vO8FBP38Ohxdh8GmmMUosHlrzHzM47LuGueymclBqiORtuRYYEGfp1HyRqIjnEX2vis1T6Lk4LPFAsLGA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:11:02.632 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:11:02.661 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:11:02.676 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:11:02.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:11:02.743 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:02.744 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:11:02.795 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:11:02.796 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2024-03-29T11:11:02.851 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:11:02.851 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:02.851 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:11:02.851 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:02.851 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:02.851 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-29 11:10:57.890000542 +0000 2024-03-29T11:11:02.851 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-29 11:08:06.258492970 +0000 2024-03-29T11:11:02.851 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-29 11:08:06.258492970 +0000 2024-03-29T11:11:02.851 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-29 11:08:06.258492970 +0000 2024-03-29T11:11:02.852 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:11:02.916 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-29T11:11:02.916 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-29T11:11:02.916 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.9026e-05 s, 5.2 MB/s 2024-03-29T11:11:02.917 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:11:02.972 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2024-03-29T11:11:03.028 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:11:03.028 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:03.028 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:11:03.028 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:03.028 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:03.028 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-29 11:10:57.890000542 +0000 2024-03-29T11:11:03.028 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-29 11:08:06.598486791 +0000 2024-03-29T11:11:03.028 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-29 11:08:06.598486791 +0000 2024-03-29T11:11:03.028 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-29 11:08:06.598486791 +0000 2024-03-29T11:11:03.029 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:11:03.093 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-29T11:11:03.094 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-29T11:11:03.094 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.551e-05 s, 5.4 MB/s 2024-03-29T11:11:03.095 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:11:03.151 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2024-03-29T11:11:03.206 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:11:03.206 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:03.206 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:11:03.206 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:03.206 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:03.206 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-29 11:10:57.891000524 +0000 2024-03-29T11:11:03.206 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-29 11:08:06.931480740 +0000 2024-03-29T11:11:03.206 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-29 11:08:06.932480721 +0000 2024-03-29T11:11:03.206 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-29 11:08:06.931480740 +0000 2024-03-29T11:11:03.207 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:11:03.271 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-29T11:11:03.272 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-29T11:11:03.272 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.4609e-05 s, 5.4 MB/s 2024-03-29T11:11:03.273 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:11:03.329 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2024-03-29T11:11:03.383 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:11:03.384 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:03.384 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:11:03.384 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:03.384 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:03.384 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-29 11:10:57.891000524 +0000 2024-03-29T11:11:03.384 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-29 11:08:07.261474743 +0000 2024-03-29T11:11:03.384 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-29 11:08:07.261474743 +0000 2024-03-29T11:11:03.384 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-29 11:08:07.261474743 +0000 2024-03-29T11:11:03.384 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:11:03.449 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-29T11:11:03.449 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-29T11:11:03.449 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000103961 s, 4.9 MB/s 2024-03-29T11:11:03.450 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:11:03.507 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:03.507 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:03.507 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:11:03.523 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:11:03.523 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-03-29T11:11:03.579 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:11:03.579 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:03.579 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:11:03.579 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:03.579 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:03.579 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-29 11:10:38.990990814 +0000 2024-03-29T11:11:03.580 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-29 11:05:56.762704967 +0000 2024-03-29T11:11:03.580 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-29 11:05:56.762704967 +0000 2024-03-29T11:11:03.580 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-29 11:05:56.762704967 +0000 2024-03-29T11:11:03.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:11:03.645 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-29T11:11:03.645 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-29T11:11:03.645 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 8.7418e-05 s, 5.9 MB/s 2024-03-29T11:11:03.647 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:11:03.703 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-03-29T11:11:03.759 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:11:03.759 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:03.759 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:11:03.759 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:03.759 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:03.759 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-29 11:10:38.991990795 +0000 2024-03-29T11:11:03.759 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-29 11:05:57.093698481 +0000 2024-03-29T11:11:03.759 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-29 11:05:57.093698481 +0000 2024-03-29T11:11:03.760 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-29 11:05:57.093698481 +0000 2024-03-29T11:11:03.760 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:11:03.829 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-29T11:11:03.829 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-29T11:11:03.829 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 7.5974e-05 s, 6.7 MB/s 2024-03-29T11:11:03.829 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:11:03.891 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-03-29T11:11:03.955 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:11:03.956 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:03.956 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:11:03.956 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:03.956 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:03.956 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-29 11:10:38.991990795 +0000 2024-03-29T11:11:03.957 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-29 11:05:57.433691819 +0000 2024-03-29T11:11:03.957 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-29 11:05:57.433691819 +0000 2024-03-29T11:11:03.957 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-29 11:05:57.433691819 +0000 2024-03-29T11:11:03.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:11:04.024 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-29T11:11:04.024 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-29T11:11:04.024 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 9.9849e-05 s, 5.1 MB/s 2024-03-29T11:11:04.026 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:11:04.082 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-03-29T11:11:04.139 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:11:04.139 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:11:04.139 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:11:04.139 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:11:04.139 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:11:04.139 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-29 11:10:38.991990795 +0000 2024-03-29T11:11:04.139 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-29 11:05:57.766685294 +0000 2024-03-29T11:11:04.139 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-29 11:05:57.766685294 +0000 2024-03-29T11:11:04.139 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-29 11:05:57.766685294 +0000 2024-03-29T11:11:04.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:11:04.207 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-29T11:11:04.207 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-29T11:11:04.207 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000101146 s, 5.1 MB/s 2024-03-29T11:11:04.208 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:11:04.270 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:04.270 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:11:04.271 INFO:tasks.ceph:Generating config... 2024-03-29T11:11:04.272 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:11:04.272 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:11:04.272 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:11:04.272 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:11:04.272 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:11:04.272 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:11:04.272 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:11:04.272 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:11:04.272 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:11:04.272 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:11:04.272 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:11:04.273 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:11:04.273 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:11:04.273 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T11:11:04.273 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:11:04.273 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:11:04.274 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:11:04.274 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:11:04.274 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:11:04.274 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:11:04.274 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:11:04.274 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:11:04.274 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:11:04.274 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:11:04.274 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:11:04.275 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:11:04.276 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:11:04.276 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:11:04.360 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:11:04.362 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:11:04.412 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:11:04.441 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.157:6789]'), ('mon.c', '[v1:172.21.15.157:6790]'), ('mon.b', '[v1:172.21.15.195:6789]')] 2024-03-29T11:11:04.441 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.157:6789],[v1:172.21.15.157:6790],[v1:172.21.15.195:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:11:04.442 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:04.442 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:11:04.499 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.157:6789]' --addv c '[v1:172.21.15.157:6790]' --addv b '[v1:172.21.15.195:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:04.584 INFO:teuthology.orchestra.run.smithi157.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:04.584 INFO:teuthology.orchestra.run.smithi157.stdout:monmaptool: generated fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:04.584 INFO:teuthology.orchestra.run.smithi157.stdout:setting min_mon_release = quincy 2024-03-29T11:11:04.584 INFO:teuthology.orchestra.run.smithi157.stdout:epoch 0 2024-03-29T11:11:04.585 INFO:teuthology.orchestra.run.smithi157.stdout:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:04.585 INFO:teuthology.orchestra.run.smithi157.stdout:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:04.585 INFO:teuthology.orchestra.run.smithi157.stdout:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:04.585 INFO:teuthology.orchestra.run.smithi157.stdout:min_mon_release 17 (quincy) 2024-03-29T11:11:04.585 INFO:teuthology.orchestra.run.smithi157.stdout:election_strategy: 1 2024-03-29T11:11:04.585 INFO:teuthology.orchestra.run.smithi157.stdout:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:04.585 INFO:teuthology.orchestra.run.smithi157.stdout:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:04.585 INFO:teuthology.orchestra.run.smithi157.stdout:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:04.585 INFO:teuthology.orchestra.run.smithi157.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:11:04.586 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:11:04.601 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a... 2024-03-29T11:11:04.602 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:11:04.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2024-03-29T11:11:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2024-03-29T11:11:04.693 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.694 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "[v1:172.21.15.157:6789],[v1:172.21.15.157:6790],[v1:172.21.15.195:6789]" 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: lockdep = True 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: mon client directed command retry = 5 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: mon election default strategy = 1 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: ms bind msgr2 = False 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: ms inject socket failures = 5000 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: ms type = async 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.695 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 600 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug trim objects = true 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.696 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2024-03-29T11:11:04.697 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore block size = 96636764160 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression algorithm = zstd 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression mode = aggressive 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore fsck on mount = True 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore zero block detection = True 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluefs = 20 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluestore = 20 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: debug monc = 20 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: debug rocksdb = 10 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd full ratio = 0.9 2024-03-29T11:11:04.698 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd class default list = * 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd class load list = * 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd client watch timeout = 120 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify cached snaps = True 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify missing on start = True 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock override recovery settings = True 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = bluestore 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue = debug_random 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue cut off = debug_random 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: osd shutdown pgref assert = True 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2024-03-29T11:11:04.699 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-29T11:11:04.700 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2024-03-29T11:11:04.700 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-03-29T11:11:04.700 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-03-29T11:11:04.700 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:11:04.700 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:11:04.701 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2024-03-29T11:11:04.701 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.701 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2024-03-29T11:11:04.701 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-29T11:11:04.701 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-03-29T11:11:04.701 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2024-03-29T11:11:04.701 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-03-29T11:11:04.701 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2024-03-29T11:11:04.701 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.702 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2024-03-29T11:11:04.703 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:11:04.703 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.703 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:11:04.703 INFO:teuthology.orchestra.run.smithi157.stdout: mon min osdmap epochs = 50 2024-03-29T11:11:04.703 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune interval = 2 2024-03-29T11:11:04.703 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune min = 15 2024-03-29T11:11:04.703 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:11:04.703 INFO:teuthology.orchestra.run.smithi157.stdout: mon scrub interval = 300 2024-03-29T11:11:04.703 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on pool no app = False 2024-03-29T11:11:04.703 INFO:teuthology.orchestra.run.smithi157.stdout: paxos service trim min = 10 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout: debug monc = 20 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout: debug objecter = 20 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout: debug rados = 20 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2024-03-29T11:11:04.704 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2024-03-29T11:11:04.705 INFO:teuthology.orchestra.run.smithi195.stdout:[global] 2024-03-29T11:11:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: chdir = "" 2024-03-29T11:11:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:11:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: auth supported = cephx 2024-03-29T11:11:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.705 INFO:teuthology.orchestra.run.smithi195.stdout: filestore xattr use omap = true 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: mon clock drift allowed = 1.000 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: osd crush chooseleaf type = 0 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: auth debug = true 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on old message = true 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on bug = true 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: mon pg warn max object skew = 0 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:11:04.706 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default size = 2 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow primary affinity = true 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow pg remap = true 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on no sortbitwise = false 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on too few osds = false 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: mon_allow_pool_size_one = true 2024-03-29T11:11:04.707 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: osd default data pool replay window = 5 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: mon allow pool delete = true 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: mon cluster log file level = debug 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: debug asserts on shutdown = true 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: mon health detail to clog = false 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: mon host = "[v1:172.21.15.157:6789],[v1:172.21.15.157:6790],[v1:172.21.15.195:6789]" 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: lockdep = True 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: mon election default strategy = 1 2024-03-29T11:11:04.708 INFO:teuthology.orchestra.run.smithi195.stdout: ms bind msgr2 = False 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 5000 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: ms type = async 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: fsid = 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout:[osd] 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: osd journal size = 100 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub load threshold = 5.0 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub max interval = 600 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: osd recover clone overlap = true 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: osd recovery max chunk = 1048576 2024-03-29T11:11:04.709 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug shutdown = true 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug op order = true 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify stray on activate = true 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug trim objects = true 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: osd open classes on start = true 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug pg log writeout = true 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: osd map max advance = 10 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: journal zero on create = true 2024-03-29T11:11:04.710 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: filestore apply finisher threads = 3 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: bdev debug aio = true 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug misdirected ops = true 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore block size = 96636764160 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore compression algorithm = zstd 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore compression mode = aggressive 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore fsck on mount = True 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore zero block detection = True 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluefs = 20 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluestore = 20 2024-03-29T11:11:04.711 INFO:teuthology.orchestra.run.smithi195.stdout: debug monc = 20 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: debug osd = 20 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: debug rocksdb = 10 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd full ratio = 0.9 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: osd class default list = * 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: osd class load list = * 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: osd client watch timeout = 120 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify cached snaps = True 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify missing on start = True 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock override recovery settings = True 2024-03-29T11:11:04.712 INFO:teuthology.orchestra.run.smithi195.stdout: osd objectstore = bluestore 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: osd op queue = debug_random 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: osd op queue cut off = debug_random 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: osd shutdown pgref assert = True 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr] 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: debug mgr = 20 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: mgr/telemetry/nag = false 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout:[mon] 2024-03-29T11:11:04.713 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: debug paxos = 20 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: mon data avail warn = 5 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: mon mgr mkfs grace = 240 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd reporter subtree level = osd 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd prime pg temp = true 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:11:04.714 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: # 1m isn't quite enough 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: mon min osdmap epochs = 50 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: mon osdmap full prune interval = 2 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: mon osdmap full prune min = 15 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: mon scrub interval = 300 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on pool no app = False 2024-03-29T11:11:04.715 INFO:teuthology.orchestra.run.smithi195.stdout: paxos service trim min = 10 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout:[client] 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: rgw cache enabled = true 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable ops log = true 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable usage log = true 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: debug monc = 20 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: debug objecter = 20 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout: debug rados = 20 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.a] 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.c] 2024-03-29T11:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.b] 2024-03-29T11:11:04.718 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:11:04.718 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:11:04.773 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:11:04.773 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:04.773 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:11:04.788 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:04.788 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:11:04.843 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T11:11:04.843 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:04.843 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:11:04.844 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:11:04.922 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:04.922 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:04.977 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T11:11:04.977 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:04.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:11:04.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:11:05.017 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:05.017 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:05.072 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:11:05.073 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:11:05.073 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:11:05.128 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:11:05.130 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:11:05.197 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:11:05.199 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:11:05.200 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:11:05.200 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:11:05.243 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:11:05.258 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:11:05.301 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:11:05.316 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:11:05.316 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:11:05.316 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:11:05.343 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:05.343 INFO:tasks.ceph:role: osd.0 2024-03-29T11:11:05.343 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T11:11:05.343 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:11:05.410 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:05.410 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:05.410 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:05.410 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:05.411 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:05.443 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-29T11:11:05.444 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-29T11:11:05.445 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:11:05.522 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:11:05.593 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:11:05.661 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:05.661 INFO:tasks.ceph:role: osd.1 2024-03-29T11:11:05.661 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T11:11:05.661 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:11:05.727 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:05.727 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:05.727 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:05.727 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:05.728 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:05.728 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:05.728 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:05.728 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:05.728 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:05.729 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:05.759 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-29T11:11:05.761 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-29T11:11:05.761 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:11:05.833 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:11:05.904 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:11:05.971 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:05.971 INFO:tasks.ceph:role: osd.2 2024-03-29T11:11:05.971 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T11:11:05.971 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:11:06.037 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:06.037 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:06.037 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:06.038 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:06.038 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:06.038 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:06.038 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:06.038 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:06.038 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:06.038 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:06.070 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-29T11:11:06.072 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-29T11:11:06.072 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:11:06.143 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:11:06.214 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:11:06.281 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:06.281 INFO:tasks.ceph:role: osd.3 2024-03-29T11:11:06.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T11:11:06.281 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:11:06.347 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:06.347 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:06.347 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:06.347 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:06.348 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:06.348 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:06.348 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:06.348 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:06.348 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:06.348 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:06.379 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-29T11:11:06.381 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-29T11:11:06.381 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:11:06.452 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:11:06.523 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:06.619 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:06.640+0000 7f861b921640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:11:06.620 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:06.640+0000 7f861b921640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:11:06.620 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:06.640+0000 7f861b921640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:11:06.620 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:06.640+0000 7f861b921640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:11:06.620 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:06.640+0000 7f861b921640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:11:06.620 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:06.641+0000 7f861b921640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:11:09.471 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:11:09.503 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:09.594 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:09.615+0000 7fb52dc1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:11:09.594 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:09.616+0000 7fb52dc1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:11:09.594 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:09.616+0000 7fb52dc1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:11:09.595 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:09.616+0000 7fb52dc1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:11:09.595 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:09.616+0000 7fb52dc1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:11:09.595 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:09.616+0000 7fb52dc1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:11:12.461 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:11:12.493 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:12.585 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:12.607+0000 7fbe45519640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:11:12.585 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:12.607+0000 7fbe45519640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:11:12.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:12.607+0000 7fbe45519640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:11:12.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:12.607+0000 7fbe45519640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:11:12.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:12.607+0000 7fbe45519640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:11:12.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:12.607+0000 7fbe45519640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:11:15.424 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:11:15.455 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:15.549 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:15.570+0000 7f1fd2412640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:11:15.549 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:15.570+0000 7f1fd2412640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:11:15.549 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:15.570+0000 7f1fd2412640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:11:15.549 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:15.570+0000 7f1fd2412640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:11:15.549 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:15.570+0000 7f1fd2412640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:11:15.549 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:15.570+0000 7f1fd2412640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:11:18.396 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:11:18.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:11:18.463 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:18.463 INFO:tasks.ceph:role: osd.4 2024-03-29T11:11:18.463 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T11:11:18.463 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:11:18.532 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:18.533 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:18.533 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:18.533 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:18.533 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:18.533 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:18.533 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:18.534 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:18.534 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:18.534 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:18.565 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-03-29T11:11:18.568 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-03-29T11:11:18.568 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:11:18.640 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:11:18.713 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:11:18.780 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:18.780 INFO:tasks.ceph:role: osd.5 2024-03-29T11:11:18.781 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T11:11:18.781 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:11:18.848 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:18.848 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:18.848 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:18.848 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:18.848 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:18.848 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:18.848 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:18.848 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:18.848 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:18.848 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:18.880 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-03-29T11:11:18.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-03-29T11:11:18.882 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:11:18.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:11:19.025 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:11:19.092 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:19.093 INFO:tasks.ceph:role: osd.6 2024-03-29T11:11:19.093 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T11:11:19.093 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:11:19.174 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:19.175 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:19.175 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:19.175 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:19.175 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:19.175 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:19.175 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:19.175 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:19.175 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:19.175 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:19.207 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-03-29T11:11:19.209 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-03-29T11:11:19.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:11:19.280 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:11:19.351 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:11:19.420 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:11:19.420 INFO:tasks.ceph:role: osd.7 2024-03-29T11:11:19.420 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T11:11:19.420 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:11:19.487 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:11:19.487 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:11:19.487 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:11:19.487 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:11:19.487 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:11:19.487 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:11:19.488 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:11:19.488 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:11:19.488 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:11:19.488 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:11:19.520 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-03-29T11:11:19.521 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-03-29T11:11:19.522 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:11:19.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:11:19.668 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:19.762 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:19.784+0000 7f10642f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:11:19.763 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:19.784+0000 7f10642f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:11:19.763 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:19.784+0000 7f10642f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:11:19.763 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:19.784+0000 7f10642f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:11:19.763 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:19.784+0000 7f10642f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:11:19.763 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:19.784+0000 7f10642f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:11:22.615 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:11:22.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:22.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:22.756+0000 7f56ebd9e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:11:22.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:22.757+0000 7f56ebd9e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:11:22.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:22.757+0000 7f56ebd9e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:11:22.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:22.757+0000 7f56ebd9e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:11:22.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:22.757+0000 7f56ebd9e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:11:22.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:22.757+0000 7f56ebd9e640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:11:25.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:11:25.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:25.715 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:25.736+0000 7f5d3768b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:11:25.715 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:25.737+0000 7f5d3768b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:11:25.715 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:25.737+0000 7f5d3768b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:11:25.715 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:25.737+0000 7f5d3768b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:11:25.715 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:25.737+0000 7f5d3768b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:11:25.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:25.737+0000 7f5d3768b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:11:28.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:11:28.603 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:28.694 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:28.716+0000 7f800ba11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:11:28.694 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:28.716+0000 7f800ba11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:11:28.694 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:28.716+0000 7f800ba11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:11:28.694 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:28.716+0000 7f800ba11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:11:28.694 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:28.716+0000 7f800ba11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:11:28.694 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:28.717+0000 7f800ba11640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:11:31.546 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:11:31.578 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:11:31.578 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:31.578 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:11:31.611 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:31.611 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:11:31.681 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:31.681 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:11:31.751 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:31.752 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:11:31.820 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:31.821 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:11:31.889 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:31.890 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:11:31.921 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:31.921 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:11:31.991 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:31.991 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:11:32.061 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:32.061 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:11:32.129 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:32.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:11:32.198 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:32.198 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:11:32.213 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:32.214 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:11:32.253 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:11:32.253 DEBUG:teuthology.orchestra.run.smithi157:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:11:32.256 DEBUG:teuthology.orchestra.run.smithi195:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:11:32.297 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.y] 2024-03-29T11:11:32.297 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDJoQZml379CBAAot7SjpT1DQKbEPOS3L3gtw== 2024-03-29T11:11:32.297 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2024-03-29T11:11:32.297 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDKoQZmIhM/JhAACnBk2nJrg8G9DO2DkT0QVQ== 2024-03-29T11:11:32.297 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDNoQZmhfjHJBAAA15zHq5sk183SVMQVzyNTw== 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDQoQZmZu0/JBAA3Dvh657Hus5HDSY6OV0NDw== 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.3] 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDToQZm6MIOIhAAi3Mh5FYWKGpT1kD/CLQSNQ== 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.x] 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDJoQZmMi8jDRAAN6Yl9wgTU9NqRc7jjyxxSA== 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.4] 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDXoQZm+e/NLhAAS5MfhiwpRGH6KbMRpo5vbg== 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.5] 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDaoQZmjjYtLRAAxruEIwWxBsd1ry+bPl6w5g== 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.6] 2024-03-29T11:11:32.298 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDdoQZmjB/7KxAA6FL6jmVghrogoNhFw/JPaQ== 2024-03-29T11:11:32.299 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.7] 2024-03-29T11:11:32.299 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDgoQZmVfi2KhAAW/Ngd7CnJQfJVTTxEs234w== 2024-03-29T11:11:32.299 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2024-03-29T11:11:32.299 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDJoQZme2ndDxAAxfbq4IoX0SbrmUjY5x7geQ== 2024-03-29T11:11:32.299 INFO:teuthology.orchestra.run.smithi157.stdout:[client.1] 2024-03-29T11:11:32.299 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDJoQZm0vdOExAAy/Qtrv4YMMCsbx7+6VDj9Q== 2024-03-29T11:11:32.320 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.y] 2024-03-29T11:11:32.320 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDJoQZml379CBAAot7SjpT1DQKbEPOS3L3gtw== 2024-03-29T11:11:32.320 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.0] 2024-03-29T11:11:32.320 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDKoQZmIhM/JhAACnBk2nJrg8G9DO2DkT0QVQ== 2024-03-29T11:11:32.320 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.1] 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDNoQZmhfjHJBAAA15zHq5sk183SVMQVzyNTw== 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.2] 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDQoQZmZu0/JBAA3Dvh657Hus5HDSY6OV0NDw== 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.3] 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDToQZm6MIOIhAAi3Mh5FYWKGpT1kD/CLQSNQ== 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.x] 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDJoQZmMi8jDRAAN6Yl9wgTU9NqRc7jjyxxSA== 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.4] 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDXoQZm+e/NLhAAS5MfhiwpRGH6KbMRpo5vbg== 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.5] 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDaoQZmjjYtLRAAxruEIwWxBsd1ry+bPl6w5g== 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.6] 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDdoQZmjB/7KxAA6FL6jmVghrogoNhFw/JPaQ== 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.7] 2024-03-29T11:11:32.321 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDgoQZmVfi2KhAAW/Ngd7CnJQfJVTTxEs234w== 2024-03-29T11:11:32.322 INFO:teuthology.orchestra.run.smithi195.stdout:[client.0] 2024-03-29T11:11:32.322 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDJoQZme2ndDxAAxfbq4IoX0SbrmUjY5x7geQ== 2024-03-29T11:11:32.322 INFO:teuthology.orchestra.run.smithi195.stdout:[client.1] 2024-03-29T11:11:32.322 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQDJoQZm0vdOExAAy/Qtrv4YMMCsbx7+6VDj9Q== 2024-03-29T11:11:32.324 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:11:32.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:11:32.416 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.419 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.470 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.524 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.527 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.578 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.633 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:11:32.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:11:32.687 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.741 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.795 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.849 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:11:32.905 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:11:32.907 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:11:32.961 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:11:32.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:11:33.019 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:11:33.019 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:11:33.047 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:11:33.143 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:11:33.170 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:11:33.237 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:11:33.333 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:11:33.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:11:33.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:11:33.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:11:33.514 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:33.517 DEBUG:teuthology.orchestra.run.smithi195:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:11:33.569 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:11:33.569 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:11:33.569 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:11:33.571 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:11:33.571 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:11:33.572 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:11:33.573 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:11:33.574 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:11:33.574 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:11:33.611 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:11:33.612 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:11:33.612 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:11:33.612 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:11:33.614 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:11:33.614 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:11:33.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:11:33.616 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:11:33.616 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:11:33.617 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:11:33.718 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.740+0000 7f17b5b72640 10 monclient: get_monmap_and_config 2024-03-29T11:11:33.718 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.740+0000 7f17b5b72640 10 monclient: build_initial_monmap 2024-03-29T11:11:33.718 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.740+0000 7f17b5b72640 10 monclient: monmap: 2024-03-29T11:11:33.718 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-29T11:11:33.718 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:33.718 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:33.741109+0000 2024-03-29T11:11:33.719 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:33.741109+0000 2024-03-29T11:11:33.719 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:33.719 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:33.719 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:33.719 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:33.719 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:33.719 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:33.720 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.740+0000 7f17b5b72640 1 Processor -- start 2024-03-29T11:11:33.720 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.740+0000 7f17b5b72640 1 -- start start 2024-03-29T11:11:33.720 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.740+0000 7f17b5b72640 10 monclient: init 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17b5b72640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17b5b72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17b5b72640 10 monclient(hunting): picked mon.noname-c con 0x7f17b00b8240 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17b5b72640 10 monclient(hunting): picked mon.noname-b con 0x7f17b00b1f60 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17b5b72640 10 monclient(hunting): picked mon.noname-a con 0x7f17b009c8c0 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17b5b72640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b009bd80 con 0x7f17b009c8c0 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17b5b72640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b009b960 con 0x7f17b00b8240 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17b5b72640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b00b2ec0 con 0x7f17b00b1f60 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17b5b72640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17b5b72640 10 monclient(hunting): authenticate will time out at 905.938301s 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17aeffd640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f17b00b1f60 0x7f17b009c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:59486/0 (socket says 172.21.15.157:59486) 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17aeffd640 1 -- 172.21.15.157:0/2574875234 learned_addr learned my addr 172.21.15.157:0/2574875234 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:33.721 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.741+0000 7f17affff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f17b009c8c0 0x7f17b014d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:40608/0 (socket says 172.21.15.157:40608) 2024-03-29T11:11:33.874 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412899517 0 0) 0x7f17b009bd80 con 0x7f17b009c8c0 2024-03-29T11:11:33.874 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:33.874 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:33.874 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:33.874 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:11:33.874 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1798003520 con 0x7f17b009c8c0 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232324828 0 0) 0x7f17b009b960 con 0x7f17b00b8240 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): my global_id is 4099 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b009bd80 con 0x7f17b00b8240 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200855992 0 0) 0x7f17b00b2ec0 con 0x7f17b00b1f60 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b009b960 con 0x7f17b00b1f60 2024-03-29T11:11:33.875 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2424275107 0 0) 0x7f17b009bd80 con 0x7f17b00b8240 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 >> v1:172.21.15.157:6790/0 conn(0x7f17b00b1f60 legacy=0x7f17b009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 >> v1:172.21.15.157:6789/0 conn(0x7f17b009c8c0 legacy=0x7f17b014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 monclient: found mon.noname-c 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b013fb60 con 0x7f17b00b8240 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f17a0001670 con 0x7f17b00b8240 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17ae7fc640 10 monclient: dump: 2024-03-29T11:11:33.876 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17b5b72640 5 monclient: authenticate success, global_id 4099 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.894+0000 7f17b5b72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17a0001850 con 0x7f17b00b8240 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17b5b72640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:33.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17ae7fc640 1 -- 172.21.15.157:0/2574875234 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f17a0001c60 con 0x7f17b00b8240 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17ae7fc640 10 monclient: dump: 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:33.878 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:33.879 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:33.879 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:33.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17b5b72640 4 set_mon_vals no callback set 2024-03-29T11:11:33.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17b5b72640 10 monclient: shutdown 2024-03-29T11:11:33.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17b5b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:33.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.895+0000 7f17b5b72640 1 -- 172.21.15.157:0/2574875234 >> v1:172.21.15.195:6789/0 conn(0x7f17b00b8240 legacy=0x7f17b013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.896+0000 7f17b5b72640 1 -- 172.21.15.157:0/2574875234 shutdown_connections 2024-03-29T11:11:33.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.896+0000 7f17b5b72640 1 -- 172.21.15.157:0/2574875234 >> 172.21.15.157:0/2574875234 conn(0x7f17b0066b00 msgr2=0x7f17b0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:33.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.896+0000 7f17b5b72640 1 -- 172.21.15.157:0/2574875234 shutdown_connections 2024-03-29T11:11:33.879 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.896+0000 7f17b5b72640 1 -- 172.21.15.157:0/2574875234 wait complete. 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.896+0000 7f17b5b72640 10 monclient: build_initial_monmap 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.897+0000 7f17b5b72640 10 monclient: monmap: 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.897+0000 7f17b5b72640 1 librados: starting msgr at 2024-03-29T11:11:33.885 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.897+0000 7f17b5b72640 1 librados: starting objecter 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.897+0000 7f17b5b72640 1 Processor -- start 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.897+0000 7f17b5b72640 1 -- start start 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.897+0000 7f17b5b72640 1 librados: setting wanted keys 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.897+0000 7f17b5b72640 1 librados: calling monclient init 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.897+0000 7f17b5b72640 10 monclient: init 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17b5b72640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17b5b72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17b5b72640 10 monclient(hunting): picked mon.noname-a con 0x7f17b009c8c0 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17b5b72640 10 monclient(hunting): picked mon.noname-c con 0x7f17b00b8240 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17b5b72640 10 monclient(hunting): picked mon.noname-b con 0x7f17b0132d90 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17b5b72640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b0133ac0 con 0x7f17b009c8c0 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17b5b72640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b0133da0 con 0x7f17b0132d90 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17b5b72640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b0134080 con 0x7f17b00b8240 2024-03-29T11:11:33.886 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17b5b72640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17aeffd640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f17b00b8240 0x7f17b01def30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:59514/0 (socket says 172.21.15.157:59514) 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17aeffd640 1 -- 172.21.15.157:0/3193142903 learned_addr learned my addr 172.21.15.157:0/3193142903 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f17b5b72640 10 monclient(hunting): authenticate will time out at 906.095660s 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f1797fff640 10 client.?.objecter ms_handle_connect 0x7f17b00b8240 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.898+0000 7f1797fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 client.?.objecter ms_handle_connect 0x7f17b0132d90 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 client.?.objecter ms_handle_connect 0x7f17b009c8c0 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842792082 0 0) 0x7f17b0133ac0 con 0x7f17b009c8c0 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:33.887 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c004eb0 con 0x7f17b009c8c0 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3151618288 0 0) 0x7f179c004eb0 con 0x7f17b009c8c0 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17b0133ac0 con 0x7f17b009c8c0 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f17a0001be0 con 0x7f17b009c8c0 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 monclient(hunting): dump: 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:33.888 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.899+0000 7f1797fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973815250 0 0) 0x7f17b0134080 con 0x7f17b00b8240 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c004eb0 con 0x7f17b00b8240 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723303445 0 0) 0x7f17b0133da0 con 0x7f17b0132d90 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:33.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17b0134080 con 0x7f17b0132d90 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 164114415 0 0) 0x7f179c004eb0 con 0x7f17b00b8240 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17b0133da0 con 0x7f17b00b8240 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f17a4001970 con 0x7f17b00b8240 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.900+0000 7f1797fff640 10 monclient(hunting): dump: 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:33.890 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3129557329 0 0) 0x7f17b0133da0 con 0x7f17b00b8240 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 >> v1:172.21.15.195:6789/0 conn(0x7f17b0132d90 legacy=0x7f17b01e2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 >> v1:172.21.15.157:6789/0 conn(0x7f17b009c8c0 legacy=0x7f17b0144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 10 monclient: _finish_hunting 0 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 1 monclient: found mon.c 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b0131a20 con 0x7f17b00b8240 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 10 monclient: _finish_auth 0 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 10 monclient: _check_auth_tickets 2024-03-29T11:11:33.891 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f17b5b72640 5 monclient: authenticate success, global_id 4115 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f17b5b72640 10 monclient: _renew_subs 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f17b5b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f17b5b72640 1 -- 172.21.15.157:0/3193142903 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b0145190 con 0x7f17b00b8240 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f17b5b72640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f17b5b72640 10 monclient: _renew_subs 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17a40047e0 con 0x7f17b00b8240 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f17b5b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f17b5b72640 1 -- 172.21.15.157:0/3193142903 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b0135690 con 0x7f17b00b8240 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f17b5b72640 1 librados: init done 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f1797fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.901+0000 7f17adffb640 4 set_mon_vals no callback set 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.902+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f17a4001d80 con 0x7f17b00b8240 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.902+0000 7f1797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:33.892 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.902+0000 7f1797fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.902+0000 7f1797fff640 10 monclient: dump: 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.902+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f17a4005040 con 0x7f17b00b8240 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.902+0000 7f17b5b72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.902+0000 7f17b5b72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:33.893 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.902+0000 7f17b5b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:33.894 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.902+0000 7f17b5b72640 1 -- 172.21.15.157:0/3193142903 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b009b170 con 0x7f17b00b8240 2024-03-29T11:11:33.894 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.903+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 478779026 0 0) 0x7f17a4005c10 con 0x7f17b00b8240 2024-03-29T11:11:33.894 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.903+0000 7f1797fff640 10 client.4115.objecter ms_dispatch 0x7f17b0000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:11:33.894 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.903+0000 7f1797fff640 3 client.4115.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:11:33.894 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.903+0000 7f1797fff640 3 client.4115.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:11:33.894 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.903+0000 7f1797fff640 20 client.4115.objecter dump_active .. 0 homeless 2024-03-29T11:11:33.894 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.905+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f17a40052f0 con 0x7f17b00b8240 2024-03-29T11:11:33.894 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.905+0000 7f1797fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:33.894 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.905+0000 7f1797fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:33.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.972+0000 7f17b5b72640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:11:33.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.972+0000 7f17b5b72640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:11:33.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.972+0000 7f17b5b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:33.951 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.972+0000 7f17b5b72640 1 -- 172.21.15.157:0/3193142903 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f17b0145380 con 0x7f17b00b8240 2024-03-29T11:11:33.954 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.976+0000 7f1797fff640 1 -- 172.21.15.157:0/3193142903 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f17a40055f0 con 0x7f17b00b8240 2024-03-29T11:11:33.954 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.976+0000 7f1797fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:11:33.954 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.976+0000 7f1797fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:11:33.954 INFO:teuthology.orchestra.run.smithi157.stderr:adjusted tunables profile to default 2024-03-29T11:11:33.955 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.976+0000 7f17b5b72640 10 librados: watch_flush enter 2024-03-29T11:11:33.955 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.976+0000 7f17b5b72640 10 librados: watch_flush exit 2024-03-29T11:11:33.955 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.976+0000 7f17b5b72640 20 client.4115.objecter shutdown clearing up homeless session... 2024-03-29T11:11:33.955 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.976+0000 7f17b5b72640 10 client.4115.objecter successfully canceled tick 2024-03-29T11:11:33.955 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.977+0000 7f17b5b72640 10 monclient: shutdown 2024-03-29T11:11:33.956 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.977+0000 7f17b5b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:33.956 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.977+0000 7f17b5b72640 1 -- 172.21.15.157:0/3193142903 >> v1:172.21.15.157:6790/0 conn(0x7f17b00b8240 legacy=0x7f17b01def30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:33.956 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.977+0000 7f17b5b72640 1 -- 172.21.15.157:0/3193142903 shutdown_connections 2024-03-29T11:11:33.956 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.977+0000 7f17b5b72640 1 -- 172.21.15.157:0/3193142903 >> 172.21.15.157:0/3193142903 conn(0x7f17b0066b00 msgr2=0x7f17b006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:33.956 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.977+0000 7f17b5b72640 1 -- 172.21.15.157:0/3193142903 shutdown_connections 2024-03-29T11:11:33.956 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.977+0000 7f17b5b72640 1 -- 172.21.15.157:0/3193142903 wait complete. 2024-03-29T11:11:33.956 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:33.977+0000 7f17b5b72640 1 librados: shutdown 2024-03-29T11:11:33.967 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:11:33.967 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:11:33.967 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:33.967 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:11:33.994 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:33.994 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:11:34.061 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:34.061 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:11:34.128 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-29T11:11:34.129 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:11:34.196 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:34.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:11:34.224 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:34.224 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:11:34.292 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:34.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:11:34.357 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:34.379+0000 7f1690c32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:11:34.359 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-29T11:11:34.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:11:34.362 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:34.384+0000 7fc60e83c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:11:34.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new fafbdcd9-9c2e-4f7a-86d5-875cb351874c 0 2024-03-29T11:11:34.411 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:34.432+0000 7f1690c32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:11:34.419 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:34.441+0000 7fc60e83c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:11:34.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.541+0000 7fdcc712b640 10 monclient: get_monmap_and_config 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.541+0000 7fdcc712b640 10 monclient: build_initial_monmap 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.541+0000 7fdcc712b640 10 monclient: monmap: 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:34.542887+0000 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:34.542887+0000 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:34.520 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.542+0000 7fdcc712b640 1 Processor -- start 2024-03-29T11:11:34.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.542+0000 7fdcc712b640 1 -- start start 2024-03-29T11:11:34.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.542+0000 7fdcc712b640 10 monclient: init 2024-03-29T11:11:34.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:34.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:34.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc009ac40 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:34.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc009c870 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:34.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc00b4360 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:34.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc009dec0 con 0x7fdcc009ac40 2024-03-29T11:11:34.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc009da30 con 0x7fdcc00b4360 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc00b39c0 con 0x7fdcc009c870 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 10 monclient(hunting): authenticate will time out at 926.814128s 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc56c8640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fdcc00b4360 0x7fdcc014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36202/0 (socket says 172.21.15.195:36202) 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc56c8640 1 -- 172.21.15.195:0/706912897 learned_addr learned my addr 172.21.15.195:0/706912897 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542129582 0 0) 0x7fdcc009da30 con 0x7fdcc00b4360 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb0003520 con 0x7fdcc00b4360 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc56c8640 1 -- 172.21.15.195:0/706912897 >> v1:172.21.15.195:6789/0 conn(0x7fdcc00b4360 legacy=0x7fdcc014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc56c8640 1 -- 172.21.15.195:0/706912897 >> v1:172.21.15.195:6789/0 conn(0x7fdcc00b4360 legacy=0x7fdcc014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:11:34.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc56c8640 1 --1- 172.21.15.195:0/706912897 >> v1:172.21.15.195:6789/0 conn(0x7fdcc00b4360 0x7fdcc014d9e0 :-1 s=OPENED pgs=14 cs=1 l=1).handle_message read tag failed 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc56c8640 1 --1- 172.21.15.195:0/706912897 >> v1:172.21.15.195:6789/0 conn(0x7fdcc00b4360 0x7fdcc014d9e0 :-1 s=OPENED pgs=14 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.195:6789/0 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188880113 0 0) 0x7fdcc00b39c0 con 0x7fdcc009c870 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc009da30 con 0x7fdcc009c870 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974198134 0 0) 0x7fdcb4001110 con 0x7fdcc009ac40 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc00b39c0 con 0x7fdcc009ac40 2024-03-29T11:11:34.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 947837939 0 0) 0x7fdcc009da30 con 0x7fdcc009c870 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 >> v1:172.21.15.195:6789/0 conn(0x7fdcc00b4360 legacy=0x7fdcc014d9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 >> v1:172.21.15.157:6789/0 conn(0x7fdcc009ac40 legacy=0x7fdcc00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 monclient: found mon.noname-b 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00b5160 con 0x7fdcc009c870 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdcac001630 con 0x7fdcc009c870 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 5 monclient: authenticate success, global_id 4118 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:34.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcbf7fe640 10 monclient: dump: 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.543+0000 7fdcc712b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcac001790 con 0x7fdcc009c870 2024-03-29T11:11:34.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcc712b640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcbf7fe640 1 -- 172.21.15.195:0/706912897 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdcac001ba0 con 0x7fdcc009c870 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcc712b640 4 set_mon_vals no callback set 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcbf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcbf7fe640 10 monclient: dump: 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.534 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcc712b640 10 monclient: shutdown 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcc712b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcc712b640 1 -- 172.21.15.195:0/706912897 >> v1:172.21.15.157:6790/0 conn(0x7fdcc009c870 legacy=0x7fdcc013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcc712b640 1 -- 172.21.15.195:0/706912897 shutdown_connections 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcc712b640 1 -- 172.21.15.195:0/706912897 >> 172.21.15.195:0/706912897 conn(0x7fdcc0066b00 msgr2=0x7fdcc0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcc712b640 1 -- 172.21.15.195:0/706912897 shutdown_connections 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.544+0000 7fdcc712b640 1 -- 172.21.15.195:0/706912897 wait complete. 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.545+0000 7fdcc712b640 10 monclient: build_initial_monmap 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.545+0000 7fdcc712b640 10 monclient: monmap: 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:34.535 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.545+0000 7fdcc712b640 1 librados: starting msgr at 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.545+0000 7fdcc712b640 1 librados: starting objecter 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.545+0000 7fdcc712b640 1 Processor -- start 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.545+0000 7fdcc712b640 1 -- start start 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.545+0000 7fdcc712b640 1 librados: setting wanted keys 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.545+0000 7fdcc712b640 1 librados: calling monclient init 2024-03-29T11:11:34.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.545+0000 7fdcc712b640 10 monclient: init 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcc712b640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcc712b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcc712b640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc009ac40 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcc712b640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc00b66a0 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcc712b640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc01d76d0 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcc712b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc00b7d10 con 0x7fdcc01d76d0 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcc712b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc01d8270 con 0x7fdcc00b66a0 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcc712b640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc00b6c90 con 0x7fdcc009ac40 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcc712b640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcc712b640 10 monclient(hunting): authenticate will time out at 926.817274s 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbffff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fdcc00b66a0 0x7fdcc01371b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36204/0 (socket says 172.21.15.195:36204) 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbffff640 1 -- 172.21.15.195:0/3844637182 learned_addr learned my addr 172.21.15.195:0/3844637182 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 client.?.objecter ms_handle_connect 0x7fdcc00b66a0 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937342985 0 0) 0x7fdcc01d8270 con 0x7fdcc00b66a0 2024-03-29T11:11:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:11:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca4004eb0 con 0x7fdcc00b66a0 2024-03-29T11:11:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 client.?.objecter ms_handle_connect 0x7fdcc01d76d0 2024-03-29T11:11:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 client.?.objecter ms_handle_connect 0x7fdcc009ac40 2024-03-29T11:11:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301700809 0 0) 0x7fdcc00b7d10 con 0x7fdcc01d76d0 2024-03-29T11:11:34.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:11:34.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc01d8270 con 0x7fdcc01d76d0 2024-03-29T11:11:34.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966168464 0 0) 0x7fdcc00b6c90 con 0x7fdcc009ac40 2024-03-29T11:11:34.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc00b7d10 con 0x7fdcc009ac40 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3295545185 0 0) 0x7fdca4004eb0 con 0x7fdcc00b66a0 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc00b6c90 con 0x7fdcc00b66a0 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdcac001be0 con 0x7fdcc00b66a0 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): dump: 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.540 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3608400286 0 0) 0x7fdcc01d8270 con 0x7fdcc01d76d0 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdca4004eb0 con 0x7fdcc01d76d0 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdcb8001af0 con 0x7fdcc01d76d0 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.546+0000 7fdcbd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 monclient(hunting): dump: 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.541 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2648481130 0 0) 0x7fdcc00b7d10 con 0x7fdcc009ac40 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc01d8270 con 0x7fdcc009ac40 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 53582596 0 0) 0x7fdcc00b6c90 con 0x7fdcc00b66a0 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 >> v1:172.21.15.157:6790/0 conn(0x7fdcc009ac40 legacy=0x7fdcc00b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 >> v1:172.21.15.157:6789/0 conn(0x7fdcc01d76d0 legacy=0x7fdcc01d7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 monclient: found mon.b 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00b77f0 con 0x7fdcc00b66a0 2024-03-29T11:11:34.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcc712b640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcc712b640 10 monclient: _renew_subs 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcc712b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcc712b640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc00b1a60 con 0x7fdcc00b66a0 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcc712b640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcc712b640 10 monclient: _renew_subs 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcc712b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcc712b640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc0137aa0 con 0x7fdcc00b66a0 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcac0038a0 con 0x7fdcc00b66a0 2024-03-29T11:11:34.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdcac003c30 con 0x7fdcc00b66a0 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 monclient: dump: 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcc712b640 1 librados: init done 2024-03-29T11:11:34.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbeffd640 4 set_mon_vals no callback set 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdcac0041a0 con 0x7fdcc00b66a0 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2100154628 0 0) 0x7fdcac006550 con 0x7fdcc00b66a0 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 10 client.4114.objecter ms_dispatch 0x7fdcc0000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.547+0000 7fdcbd7fa640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.548+0000 7fdcc712b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.548+0000 7fdcc712b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.548+0000 7fdcc712b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.548+0000 7fdcc712b640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc009b020 con 0x7fdcc00b66a0 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.550+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdcc0137aa0 con 0x7fdcc00b66a0 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.550+0000 7fdcbd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.545 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.550+0000 7fdcbd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:34.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.632+0000 7fdcc712b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fafbdcd9-9c2e-4f7a-86d5-875cb351874c", "id": 0}] 2024-03-29T11:11:34.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.632+0000 7fdcc712b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fafbdcd9-9c2e-4f7a-86d5-875cb351874c", "id": 0}] 2024-03-29T11:11:34.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.632+0000 7fdcc712b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:34.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.632+0000 7fdcc712b640 1 -- 172.21.15.195:0/3844637182 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fafbdcd9-9c2e-4f7a-86d5-875cb351874c", "id": 0} v 0) v1 -- 0x7fdcc0144bb0 con 0x7fdcc00b66a0 2024-03-29T11:11:34.612 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.634+0000 7fdcbd7fa640 1 -- 172.21.15.195:0/3844637182 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fafbdcd9-9c2e-4f7a-86d5-875cb351874c", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2697645221 0 732707515) 0x7fdcac006bb0 con 0x7fdcc00b66a0 2024-03-29T11:11:34.612 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.634+0000 7fdcbd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fafbdcd9-9c2e-4f7a-86d5-875cb351874c", "id": 0}] 2024-03-29T11:11:34.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.634+0000 7fdcbd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:34.613 INFO:teuthology.orchestra.run.smithi195.stdout:0 2024-03-29T11:11:34.614 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.635+0000 7fdcc712b640 10 librados: watch_flush enter 2024-03-29T11:11:34.614 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.635+0000 7fdcc712b640 10 librados: watch_flush exit 2024-03-29T11:11:34.614 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.635+0000 7fdcc712b640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T11:11:34.614 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.635+0000 7fdcc712b640 10 client.4114.objecter successfully canceled tick 2024-03-29T11:11:34.614 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.635+0000 7fdcc712b640 10 monclient: shutdown 2024-03-29T11:11:34.615 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.635+0000 7fdcc712b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:34.615 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.635+0000 7fdcc712b640 1 -- 172.21.15.195:0/3844637182 >> v1:172.21.15.195:6789/0 conn(0x7fdcc00b66a0 legacy=0x7fdcc01371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.615 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.636+0000 7fdcc712b640 1 -- 172.21.15.195:0/3844637182 shutdown_connections 2024-03-29T11:11:34.615 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.636+0000 7fdcc712b640 1 -- 172.21.15.195:0/3844637182 >> 172.21.15.195:0/3844637182 conn(0x7fdcc0066b00 msgr2=0x7fdcc013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.615 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.636+0000 7fdcc712b640 1 -- 172.21.15.195:0/3844637182 shutdown_connections 2024-03-29T11:11:34.615 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.636+0000 7fdcc712b640 1 -- 172.21.15.195:0/3844637182 wait complete. 2024-03-29T11:11:34.615 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.636+0000 7fdcc712b640 1 librados: shutdown 2024-03-29T11:11:34.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 9cc8713a-d703-408a-9b9d-98b8e5aace07 1 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.745+0000 7f946a686640 10 monclient: get_monmap_and_config 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.745+0000 7f946a686640 10 monclient: build_initial_monmap 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.745+0000 7f946a686640 10 monclient: monmap: 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:34.746956+0000 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:34.746956+0000 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:34.724 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.734 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.746+0000 7f946a686640 1 Processor -- start 2024-03-29T11:11:34.734 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.746+0000 7f946a686640 1 -- start start 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.746+0000 7f946a686640 10 monclient: init 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 10 monclient(hunting): picked mon.noname-c con 0x7f946409ac40 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 10 monclient(hunting): picked mon.noname-b con 0x7f946409c870 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 10 monclient(hunting): picked mon.noname-a con 0x7f94640b4360 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946409dec0 con 0x7f94640b4360 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946409da30 con 0x7f946409ac40 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94640b39c0 con 0x7f946409c870 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 10 monclient(hunting): authenticate will time out at 927.018269s 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9463fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f946409ac40 0x7f94640b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36208/0 (socket says 172.21.15.195:36208) 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9463fff640 1 -- 172.21.15.195:0/1293021957 learned_addr learned my addr 172.21.15.195:0/1293021957 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:34.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f94637fe640 1 -- 172.21.15.195:0/1293021957 >> v1:172.21.15.157:6790/0 conn(0x7f946409c870 legacy=0x7f946413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f94637fe640 1 -- 172.21.15.195:0/1293021957 >> v1:172.21.15.157:6790/0 conn(0x7f946409c870 legacy=0x7f946413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f94637fe640 1 --1- 172.21.15.195:0/1293021957 >> v1:172.21.15.157:6790/0 conn(0x7f946409c870 0x7f946413f3a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300341323 0 0) 0x7f946409da30 con 0x7f946409ac40 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f944c003520 con 0x7f946409ac40 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251007091 0 0) 0x7f945c001490 con 0x7f94640b4360 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:11:34.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946409da30 con 0x7f94640b4360 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1293746166 0 0) 0x7f944c003520 con 0x7f946409ac40 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 >> v1:172.21.15.157:6790/0 conn(0x7f946409c870 legacy=0x7f946413f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 >> v1:172.21.15.157:6789/0 conn(0x7f94640b4360 legacy=0x7f946414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 1 monclient: found mon.noname-c 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94640b5160 con 0x7f946409ac40 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient: _finish_auth 0 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9458001690 con 0x7f946409ac40 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:34.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f9462ffd640 10 monclient: dump: 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.747+0000 7f946a686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:34.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9458001870 con 0x7f946409ac40 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f9462ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f946a686640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f946a686640 4 set_mon_vals no callback set 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f9462ffd640 1 -- 172.21.15.195:0/1293021957 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9458001c80 con 0x7f946409ac40 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f9462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f9462ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f9462ffd640 10 monclient: dump: 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.739 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f946a686640 10 monclient: shutdown 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f946a686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f946a686640 1 -- 172.21.15.195:0/1293021957 >> v1:172.21.15.195:6789/0 conn(0x7f946409ac40 legacy=0x7f94640b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f946a686640 1 -- 172.21.15.195:0/1293021957 shutdown_connections 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f946a686640 1 -- 172.21.15.195:0/1293021957 >> 172.21.15.195:0/1293021957 conn(0x7f9464066b00 msgr2=0x7f9464066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f946a686640 1 -- 172.21.15.195:0/1293021957 shutdown_connections 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.748+0000 7f946a686640 1 -- 172.21.15.195:0/1293021957 wait complete. 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.749+0000 7f946a686640 10 monclient: build_initial_monmap 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.749+0000 7f946a686640 10 monclient: monmap: 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.749+0000 7f946a686640 1 librados: starting msgr at 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.749+0000 7f946a686640 1 librados: starting objecter 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.749+0000 7f946a686640 1 Processor -- start 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.749+0000 7f946a686640 1 -- start start 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.749+0000 7f946a686640 1 librados: setting wanted keys 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.749+0000 7f946a686640 1 librados: calling monclient init 2024-03-29T11:11:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.749+0000 7f946a686640 10 monclient: init 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f946a686640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f946a686640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f946a686640 10 monclient(hunting): picked mon.noname-b con 0x7f946409ac40 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f946a686640 10 monclient(hunting): picked mon.noname-a con 0x7f9464139bf0 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f946a686640 10 monclient(hunting): picked mon.noname-c con 0x7f94641406b0 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f946a686640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9464141bd0 con 0x7f9464139bf0 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f946a686640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94640b7f40 con 0x7f946409ac40 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f946a686640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9464142ce0 con 0x7f94641406b0 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f946a686640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f946a686640 10 monclient(hunting): authenticate will time out at 927.021344s 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9463fff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f946409ac40 0x7f94641425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36220/0 (socket says 172.21.15.195:36220) 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9463fff640 1 -- 172.21.15.195:0/338059958 learned_addr learned my addr 172.21.15.195:0/338059958 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:34.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 client.?.objecter ms_handle_connect 0x7f946409ac40 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 client.?.objecter ms_handle_connect 0x7f9464139bf0 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567856401 0 0) 0x7f94640b7f40 con 0x7f946409ac40 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9450004eb0 con 0x7f946409ac40 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 client.?.objecter ms_handle_connect 0x7f94641406b0 2024-03-29T11:11:34.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9463fff640 1 -- 172.21.15.195:0/338059958 >> v1:172.21.15.195:6789/0 conn(0x7f946409ac40 legacy=0x7f94641425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9463fff640 1 -- 172.21.15.195:0/338059958 >> v1:172.21.15.195:6789/0 conn(0x7f946409ac40 legacy=0x7f94641425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9463fff640 1 --1- 172.21.15.195:0/338059958 >> v1:172.21.15.195:6789/0 conn(0x7f946409ac40 0x7f94641425d0 :-1 s=OPENED pgs=17 cs=1 l=1).handle_message read tag failed 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9463fff640 1 --1- 172.21.15.195:0/338059958 >> v1:172.21.15.195:6789/0 conn(0x7f946409ac40 0x7f94641425d0 :-1 s=OPENED pgs=17 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.195:6789/0 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811794486 0 0) 0x7f9464141bd0 con 0x7f9464139bf0 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94640b7f40 con 0x7f9464139bf0 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993506976 0 0) 0x7f9464142ce0 con 0x7f94641406b0 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9464141bd0 con 0x7f94641406b0 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2745555277 0 0) 0x7f94640b7f40 con 0x7f9464139bf0 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.750+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9464142ce0 con 0x7f9464139bf0 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9454003f70 con 0x7f9464139bf0 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.745 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1996629598 0 0) 0x7f9464141bd0 con 0x7f94641406b0 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94640b7f40 con 0x7f94641406b0 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f945c001b80 con 0x7f94641406b0 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.746 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1664356312 0 0) 0x7f9464142ce0 con 0x7f9464139bf0 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 >> v1:172.21.15.157:6790/0 conn(0x7f94641406b0 legacy=0x7f9464138ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 >> v1:172.21.15.195:6789/0 conn(0x7f946409ac40 legacy=0x7f94641425d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 monclient: found mon.a 2024-03-29T11:11:34.747 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9464142e70 con 0x7f9464139bf0 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient: _finish_auth 0 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f946a686640 5 monclient: authenticate success, global_id 4125 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f946a686640 10 monclient: _renew_subs 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f946a686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f946a686640 1 -- 172.21.15.195:0/338059958 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9464139600 con 0x7f9464139bf0 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f946a686640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f946a686640 10 monclient: _renew_subs 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f946a686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f946a686640 1 -- 172.21.15.195:0/338059958 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94641e38e0 con 0x7f9464139bf0 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f946a686640 1 librados: init done 2024-03-29T11:11:34.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9454000ed0 con 0x7f9464139bf0 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f94540044e0 con 0x7f9464139bf0 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f94627fc640 4 set_mon_vals no callback set 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 monclient: dump: 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.749 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f94540016f0 con 0x7f9464139bf0 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 414725596 0 0) 0x7f9454006430 con 0x7f9464139bf0 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 10 client.4125.objecter ms_dispatch 0x7f9464000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.751+0000 7f9460ff9640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.752+0000 7f946a686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.752+0000 7f946a686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.752+0000 7f946a686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.752+0000 7f946a686640 1 -- 172.21.15.195:0/338059958 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9444005080 con 0x7f9464139bf0 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.754+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9454001b80 con 0x7f9464139bf0 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.754+0000 7f9460ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.750 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.754+0000 7f9460ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:34.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.826+0000 7f946a686640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9cc8713a-d703-408a-9b9d-98b8e5aace07", "id": 1}] 2024-03-29T11:11:34.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.826+0000 7f946a686640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9cc8713a-d703-408a-9b9d-98b8e5aace07", "id": 1}] 2024-03-29T11:11:34.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.826+0000 7f946a686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:11:34.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.826+0000 7f946a686640 1 -- 172.21.15.195:0/338059958 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9cc8713a-d703-408a-9b9d-98b8e5aace07", "id": 1} v 0) v1 -- 0x7f94440058b0 con 0x7f9464139bf0 2024-03-29T11:11:34.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.828+0000 7f9460ff9640 1 -- 172.21.15.195:0/338059958 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9cc8713a-d703-408a-9b9d-98b8e5aace07", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2179566850 0 940483276) 0x7f94540057c0 con 0x7f9464139bf0 2024-03-29T11:11:34.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.828+0000 7f9460ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9cc8713a-d703-408a-9b9d-98b8e5aace07", "id": 1}] 2024-03-29T11:11:34.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.828+0000 7f9460ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:34.806 INFO:teuthology.orchestra.run.smithi195.stdout:1 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 10 librados: watch_flush enter 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 10 librados: watch_flush exit 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 10 client.4125.objecter successfully canceled tick 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 10 monclient: shutdown 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 1 -- 172.21.15.195:0/338059958 >> v1:172.21.15.157:6789/0 conn(0x7f9464139bf0 legacy=0x7f946413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 1 -- 172.21.15.195:0/338059958 shutdown_connections 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 1 -- 172.21.15.195:0/338059958 >> 172.21.15.195:0/338059958 conn(0x7f9464066b00 msgr2=0x7f946413cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 1 -- 172.21.15.195:0/338059958 shutdown_connections 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 1 -- 172.21.15.195:0/338059958 wait complete. 2024-03-29T11:11:34.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.829+0000 7f946a686640 1 librados: shutdown 2024-03-29T11:11:34.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 7f55dd07-77ef-4e1b-9463-6e172452c432 2 2024-03-29T11:11:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.941+0000 7f73aeb32640 10 monclient: get_monmap_and_config 2024-03-29T11:11:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.941+0000 7f73aeb32640 10 monclient: build_initial_monmap 2024-03-29T11:11:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.941+0000 7f73aeb32640 10 monclient: monmap: 2024-03-29T11:11:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:34.942135+0000 2024-03-29T11:11:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:34.942135+0000 2024-03-29T11:11:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:34.919 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.920 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:34.920 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:34.920 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:34.920 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.930 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.941+0000 7f73aeb32640 1 Processor -- start 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.941+0000 7f73aeb32640 1 -- start start 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.941+0000 7f73aeb32640 10 monclient: init 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73aeb32640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73aeb32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73aeb32640 10 monclient(hunting): picked mon.noname-a con 0x7f73a809ac40 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73aeb32640 10 monclient(hunting): picked mon.noname-b con 0x7f73a809c870 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73aeb32640 10 monclient(hunting): picked mon.noname-c con 0x7f73a80b4360 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73aeb32640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a809dec0 con 0x7f73a809ac40 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73aeb32640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a809da30 con 0x7f73a80b4360 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73aeb32640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a80b39c0 con 0x7f73a809c870 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73aeb32640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:34.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73aeb32640 10 monclient(hunting): authenticate will time out at 927.213409s 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73ad0cf640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f73a80b4360 0x7f73a814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36224/0 (socket says 172.21.15.195:36224) 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73ad0cf640 1 -- 172.21.15.195:0/4046717319 learned_addr learned my addr 172.21.15.195:0/4046717319 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832430097 0 0) 0x7f73a809da30 con 0x7f73a80b4360 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7398003520 con 0x7f73a80b4360 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153862902 0 0) 0x7f73a80b39c0 con 0x7f73a809c870 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a809da30 con 0x7f73a809c870 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519761132 0 0) 0x7f739c001110 con 0x7f73a809ac40 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.942+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a80b39c0 con 0x7f73a809ac40 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154554001 0 0) 0x7f73a809da30 con 0x7f73a809c870 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 >> v1:172.21.15.195:6789/0 conn(0x7f73a80b4360 legacy=0x7f73a814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 >> v1:172.21.15.157:6789/0 conn(0x7f73a809ac40 legacy=0x7f73a80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:34.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 1 monclient: found mon.noname-b 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a80b5160 con 0x7f73a809c870 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 5 monclient: authenticate success, global_id 4127 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7394001670 con 0x7f73a809c870 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: dump: 2024-03-29T11:11:34.934 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73940017d0 con 0x7f73a809c870 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:34.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 1 -- 172.21.15.195:0/4046717319 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7394001c20 con 0x7f73a809c870 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 4 set_mon_vals no callback set 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73a77fe640 10 monclient: dump: 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.936 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 10 monclient: shutdown 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 1 -- 172.21.15.195:0/4046717319 >> v1:172.21.15.157:6790/0 conn(0x7f73a809c870 legacy=0x7f73a813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 1 -- 172.21.15.195:0/4046717319 shutdown_connections 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 1 -- 172.21.15.195:0/4046717319 >> 172.21.15.195:0/4046717319 conn(0x7f73a8066b00 msgr2=0x7f73a8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 1 -- 172.21.15.195:0/4046717319 shutdown_connections 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.943+0000 7f73aeb32640 1 -- 172.21.15.195:0/4046717319 wait complete. 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.944+0000 7f73aeb32640 10 monclient: build_initial_monmap 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.944+0000 7f73aeb32640 10 monclient: monmap: 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-29T11:11:34.937 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:34.938 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.944+0000 7f73aeb32640 1 librados: starting msgr at 2024-03-29T11:11:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.944+0000 7f73aeb32640 1 librados: starting objecter 2024-03-29T11:11:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.944+0000 7f73aeb32640 1 Processor -- start 2024-03-29T11:11:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.944+0000 7f73aeb32640 1 -- start start 2024-03-29T11:11:34.938 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.944+0000 7f73aeb32640 1 librados: setting wanted keys 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.944+0000 7f73aeb32640 1 librados: calling monclient init 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.944+0000 7f73aeb32640 10 monclient: init 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73aeb32640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73aeb32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73aeb32640 10 monclient(hunting): picked mon.noname-a con 0x7f73a809ac40 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73aeb32640 10 monclient(hunting): picked mon.noname-c con 0x7f73a80b0b00 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73aeb32640 10 monclient(hunting): picked mon.noname-b con 0x7f73a81df6b0 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73aeb32640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a80ad5e0 con 0x7f73a809ac40 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73aeb32640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a80ad770 con 0x7f73a81df6b0 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73aeb32640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a80aff80 con 0x7f73a80b0b00 2024-03-29T11:11:34.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73aeb32640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73aeb32640 10 monclient(hunting): authenticate will time out at 927.216617s 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73ad0cf640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f73a81df6b0 0x7f73a81e1aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36226/0 (socket says 172.21.15.195:36226) 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73ad0cf640 1 -- 172.21.15.195:0/3410101367 learned_addr learned my addr 172.21.15.195:0/3410101367 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73ac8ce640 1 --1- 172.21.15.195:0/3410101367 >> v1:172.21.15.157:6789/0 conn(0x7f73a809ac40 0x7f73a81da770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.195:60852/0 (socket says 172.21.15.195:60852) 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73a57fa640 10 client.?.objecter ms_handle_connect 0x7f73a81df6b0 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73a57fa640 10 client.?.objecter ms_handle_connect 0x7f73a809ac40 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73a57fa640 10 client.?.objecter ms_handle_connect 0x7f73a80b0b00 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.945+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278180259 0 0) 0x7f73a80ad770 con 0x7f73a81df6b0 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f738c004eb0 con 0x7f73a81df6b0 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255305613 0 0) 0x7f73a80ad5e0 con 0x7f73a809ac40 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a80ad770 con 0x7f73a809ac40 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386922707 0 0) 0x7f73a80aff80 con 0x7f73a80b0b00 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a80ad5e0 con 0x7f73a80b0b00 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 578403777 0 0) 0x7f73a80ad5e0 con 0x7f73a80b0b00 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73a80aff80 con 0x7f73a80b0b00 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7394001ae0 con 0x7f73a80b0b00 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): dump: 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.942 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.943 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3491138693 0 0) 0x7f738c004eb0 con 0x7f73a81df6b0 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73a80ad5e0 con 0x7f73a81df6b0 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1389633779 0 0) 0x7f73a80ad770 con 0x7f73a809ac40 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f738c004eb0 con 0x7f73a809ac40 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73a0001bd0 con 0x7f73a81df6b0 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): dump: 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.944 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f739c001900 con 0x7f73a809ac40 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient(hunting): dump: 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.945 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1326650480 0 0) 0x7f73a80aff80 con 0x7f73a80b0b00 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 >> v1:172.21.15.195:6789/0 conn(0x7f73a81df6b0 legacy=0x7f73a81e1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 >> v1:172.21.15.157:6789/0 conn(0x7f73a809ac40 legacy=0x7f73a81da770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 monclient: found mon.c 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:34.946 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a80b0110 con 0x7f73a80b0b00 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient: _finish_auth 0 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73aeb32640 5 monclient: authenticate success, global_id 4130 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73aeb32640 10 monclient: _renew_subs 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73aeb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73aeb32640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a81dae80 con 0x7f73a80b0b00 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73aeb32640 10 client.4130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73aeb32640 10 monclient: _renew_subs 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73aeb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73aeb32640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a81e33a0 con 0x7f73a80b0b00 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73aeb32640 1 librados: init done 2024-03-29T11:11:34.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7394001210 con 0x7f73a80b0b00 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7394003a40 con 0x7f73a80b0b00 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a6ffd640 4 set_mon_vals no callback set 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 monclient: dump: 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:34.948 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7394003fb0 con 0x7f73a80b0b00 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1436712514 0 0) 0x7f7394005600 con 0x7f73a80b0b00 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 10 client.4130.objecter ms_dispatch 0x7f73a8000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 3 client.4130.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 3 client.4130.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.946+0000 7f73a57fa640 20 client.4130.objecter dump_active .. 0 homeless 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.947+0000 7f73aeb32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.947+0000 7f73aeb32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.947+0000 7f73aeb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.947+0000 7f73aeb32640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7388005080 con 0x7f73a80b0b00 2024-03-29T11:11:34.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.949+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7394005290 con 0x7f73a80b0b00 2024-03-29T11:11:34.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.949+0000 7f73a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:34.949+0000 7f73a57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:34.950 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:34.968+0000 7f1690c32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:11:34.979 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.000+0000 7fc60e83c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:11:35.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.022+0000 7f73aeb32640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7f55dd07-77ef-4e1b-9463-6e172452c432", "id": 2}] 2024-03-29T11:11:35.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.022+0000 7f73aeb32640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7f55dd07-77ef-4e1b-9463-6e172452c432", "id": 2}] 2024-03-29T11:11:35.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.022+0000 7f73aeb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:35.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.022+0000 7f73aeb32640 1 -- 172.21.15.195:0/3410101367 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd new", "uuid": "7f55dd07-77ef-4e1b-9463-6e172452c432", "id": 2} v 0) v1 -- 0x7f73880058b0 con 0x7f73a80b0b00 2024-03-29T11:11:35.001 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:35.023+0000 7f1690c32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:11:35.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.026+0000 7f73a57fa640 1 -- 172.21.15.195:0/3410101367 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f55dd07-77ef-4e1b-9463-6e172452c432", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 966410382 0 216599125) 0x7f7394005890 con 0x7f73a80b0b00 2024-03-29T11:11:35.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.026+0000 7f73a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7f55dd07-77ef-4e1b-9463-6e172452c432", "id": 2}] 2024-03-29T11:11:35.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.026+0000 7f73a57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:35.004 INFO:teuthology.orchestra.run.smithi195.stdout:2 2024-03-29T11:11:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.026+0000 7f73aeb32640 10 librados: watch_flush enter 2024-03-29T11:11:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.026+0000 7f73aeb32640 10 librados: watch_flush exit 2024-03-29T11:11:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.026+0000 7f73aeb32640 20 client.4130.objecter shutdown clearing up homeless session... 2024-03-29T11:11:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.026+0000 7f73aeb32640 10 client.4130.objecter successfully canceled tick 2024-03-29T11:11:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.026+0000 7f73aeb32640 10 monclient: shutdown 2024-03-29T11:11:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.026+0000 7f73aeb32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:35.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.026+0000 7f73aeb32640 1 -- 172.21.15.195:0/3410101367 >> v1:172.21.15.157:6790/0 conn(0x7f73a80b0b00 legacy=0x7f73a80b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.027+0000 7f73aeb32640 1 -- 172.21.15.195:0/3410101367 shutdown_connections 2024-03-29T11:11:35.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.027+0000 7f73aeb32640 1 -- 172.21.15.195:0/3410101367 >> 172.21.15.195:0/3410101367 conn(0x7f73a8066b00 msgr2=0x7f73a813e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.027+0000 7f73aeb32640 1 -- 172.21.15.195:0/3410101367 shutdown_connections 2024-03-29T11:11:35.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.027+0000 7f73aeb32640 1 -- 172.21.15.195:0/3410101367 wait complete. 2024-03-29T11:11:35.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.027+0000 7f73aeb32640 1 librados: shutdown 2024-03-29T11:11:35.015 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 5315b994-aa27-42bb-84d2-00a227ac264e 3 2024-03-29T11:11:35.036 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.058+0000 7fc60e83c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:11:35.047 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:35.069+0000 7f1690c32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:11:35.084 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.105+0000 7fc60e83c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:11:35.101 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:35.123+0000 7f1690c32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:11:35.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.137+0000 7feabcd04640 10 monclient: get_monmap_and_config 2024-03-29T11:11:35.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.137+0000 7feabcd04640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.137+0000 7feabcd04640 10 monclient: monmap: 2024-03-29T11:11:35.115 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:35.138200+0000 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:35.138200+0000 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.137+0000 7feabcd04640 1 Processor -- start 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.137+0000 7feabcd04640 1 -- start start 2024-03-29T11:11:35.116 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.137+0000 7feabcd04640 10 monclient: init 2024-03-29T11:11:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feabcd04640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feabcd04640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feabcd04640 10 monclient(hunting): picked mon.noname-c con 0x7feab809ac40 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feabcd04640 10 monclient(hunting): picked mon.noname-b con 0x7feab809c870 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feabcd04640 10 monclient(hunting): picked mon.noname-a con 0x7feab80b4360 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feabcd04640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab809dec0 con 0x7feab80b4360 2024-03-29T11:11:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feabcd04640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab809da30 con 0x7feab809ac40 2024-03-29T11:11:35.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feabcd04640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab80b39c0 con 0x7feab809c870 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feabcd04640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feabcd04640 10 monclient(hunting): authenticate will time out at 927.409482s 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feab659c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7feab809ac40 0x7feab80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36238/0 (socket says 172.21.15.195:36238) 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feab659c640 1 -- 172.21.15.195:0/2196095147 learned_addr learned my addr 172.21.15.195:0/2196095147 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659269575 0 0) 0x7feab809da30 con 0x7feab809ac40 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feab559a640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feab559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feab559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feab559a640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa0003520 con 0x7feab809ac40 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.138+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203970837 0 0) 0x7feab80b39c0 con 0x7feab809c870 2024-03-29T11:11:35.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:11:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab809da30 con 0x7feab809c870 2024-03-29T11:11:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930088195 0 0) 0x7feaa8001490 con 0x7feab80b4360 2024-03-29T11:11:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:11:35.128 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab80b39c0 con 0x7feab80b4360 2024-03-29T11:11:35.128 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3480332601 0 0) 0x7feaa0003520 con 0x7feab809ac40 2024-03-29T11:11:35.128 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 >> v1:172.21.15.157:6790/0 conn(0x7feab809c870 legacy=0x7feab813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.128 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 >> v1:172.21.15.157:6789/0 conn(0x7feab80b4360 legacy=0x7feab814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 monclient: found mon.noname-c 2024-03-29T11:11:35.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80b5160 con 0x7feab809ac40 2024-03-29T11:11:35.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feab0001690 con 0x7feab809ac40 2024-03-29T11:11:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feabcd04640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:11:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feabcd04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: dump: 2024-03-29T11:11:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.131 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feabcd04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feab0001870 con 0x7feab809ac40 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 1 -- 172.21.15.195:0/2196095147 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feab0001c80 con 0x7feab809ac40 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feabcd04640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feabcd04640 4 set_mon_vals no callback set 2024-03-29T11:11:35.132 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feab559a640 10 monclient: dump: 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feabcd04640 10 monclient: shutdown 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feabcd04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:35.133 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feabcd04640 1 -- 172.21.15.195:0/2196095147 >> v1:172.21.15.195:6789/0 conn(0x7feab809ac40 legacy=0x7feab80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.139+0000 7feabcd04640 1 -- 172.21.15.195:0/2196095147 shutdown_connections 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.140+0000 7feabcd04640 1 -- 172.21.15.195:0/2196095147 >> 172.21.15.195:0/2196095147 conn(0x7feab8066b00 msgr2=0x7feab8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.140+0000 7feabcd04640 1 -- 172.21.15.195:0/2196095147 shutdown_connections 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.140+0000 7feabcd04640 1 -- 172.21.15.195:0/2196095147 wait complete. 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.140+0000 7feabcd04640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.140+0000 7feabcd04640 10 monclient: monmap: 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.134 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 1 librados: starting msgr at 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 1 librados: starting objecter 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 1 Processor -- start 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 1 -- start start 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 1 librados: setting wanted keys 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 1 librados: calling monclient init 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 10 monclient: init 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 10 monclient(hunting): picked mon.noname-b con 0x7feab809ac40 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 10 monclient(hunting): picked mon.noname-a con 0x7feab80acf60 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:35.135 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 10 monclient(hunting): picked mon.noname-c con 0x7feab80b0800 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab81da7b0 con 0x7feab80acf60 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab80b0c70 con 0x7feab809ac40 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab81db350 con 0x7feab80b0800 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feabcd04640 10 monclient(hunting): authenticate will time out at 927.412741s 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feab659c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7feab809ac40 0x7feab80af9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36246/0 (socket says 172.21.15.195:36246) 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feab659c640 1 -- 172.21.15.195:0/1307115786 learned_addr learned my addr 172.21.15.195:0/1307115786 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feaa6ffd640 10 client.?.objecter ms_handle_connect 0x7feab809ac40 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.141+0000 7feaa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 client.?.objecter ms_handle_connect 0x7feab80acf60 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 client.?.objecter ms_handle_connect 0x7feab80b0800 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469388969 0 0) 0x7feab80b0c70 con 0x7feab809ac40 2024-03-29T11:11:35.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea98004eb0 con 0x7feab809ac40 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134581736 0 0) 0x7feab81db350 con 0x7feab80b0800 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab80b0c70 con 0x7feab80b0800 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144592428 0 0) 0x7feab81da7b0 con 0x7feab80acf60 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab81db350 con 0x7feab80acf60 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1718088845 0 0) 0x7fea98004eb0 con 0x7feab809ac40 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feab81da7b0 con 0x7feab809ac40 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feab0001bc0 con 0x7feab809ac40 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): dump: 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.138 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 390781159 0 0) 0x7feab80b0c70 con 0x7feab80b0800 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea98004eb0 con 0x7feab80b0800 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feaa8001b40 con 0x7feab80b0800 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): dump: 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.139 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438936028 0 0) 0x7feab81db350 con 0x7feab80acf60 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feab80b0c70 con 0x7feab80acf60 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feaac0018b0 con 0x7feab80acf60 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient(hunting): dump: 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.140 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 898985175 0 0) 0x7feab81da7b0 con 0x7feab809ac40 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 >> v1:172.21.15.157:6790/0 conn(0x7feab80b0800 legacy=0x7feab81e1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 >> v1:172.21.15.157:6789/0 conn(0x7feab80acf60 legacy=0x7feab81dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 monclient: found mon.b 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab81db4e0 con 0x7feab809ac40 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feabcd04640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feabcd04640 10 monclient: _renew_subs 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feabcd04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feabcd04640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab81e3240 con 0x7feab809ac40 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feabcd04640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feabcd04640 10 monclient: _renew_subs 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feabcd04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feabcd04640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feab81e35b0 con 0x7feab809ac40 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feabcd04640 1 librados: init done 2024-03-29T11:11:35.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feab0004bf0 con 0x7feab809ac40 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feab0004d60 con 0x7feab809ac40 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 monclient: dump: 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.143 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feab4d99640 4 set_mon_vals no callback set 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7feab0005250 con 0x7feab809ac40 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1207462002 0 0) 0x7feab0001170 con 0x7feab809ac40 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 10 client.4132.objecter ms_dispatch 0x7feab8000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.142+0000 7feaa6ffd640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.143+0000 7feabcd04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.143+0000 7feabcd04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.143+0000 7feabcd04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.143+0000 7feabcd04640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9c005080 con 0x7feab809ac40 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.145+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7feab0005b10 con 0x7feab809ac40 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.145+0000 7feaa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.145+0000 7feaa6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:35.146 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.162+0000 7fc60e83c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:11:35.147 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:35.166+0000 7f1690c32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:11:35.184 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.206+0000 7fc60e83c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:11:35.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.218+0000 7feabcd04640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5315b994-aa27-42bb-84d2-00a227ac264e", "id": 3}] 2024-03-29T11:11:35.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.218+0000 7feabcd04640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5315b994-aa27-42bb-84d2-00a227ac264e", "id": 3}] 2024-03-29T11:11:35.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.218+0000 7feabcd04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.218+0000 7feabcd04640 1 -- 172.21.15.195:0/1307115786 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5315b994-aa27-42bb-84d2-00a227ac264e", "id": 3} v 0) v1 -- 0x7fea9c0058b0 con 0x7feab809ac40 2024-03-29T11:11:35.198 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.221+0000 7feaa6ffd640 1 -- 172.21.15.195:0/1307115786 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5315b994-aa27-42bb-84d2-00a227ac264e", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1195620045 0 525046306) 0x7feab00055d0 con 0x7feab809ac40 2024-03-29T11:11:35.199 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.221+0000 7feaa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5315b994-aa27-42bb-84d2-00a227ac264e", "id": 3}] 2024-03-29T11:11:35.199 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.221+0000 7feaa6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:35.199 INFO:teuthology.orchestra.run.smithi195.stdout:3 2024-03-29T11:11:35.200 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.221+0000 7feabcd04640 10 librados: watch_flush enter 2024-03-29T11:11:35.200 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.221+0000 7feabcd04640 10 librados: watch_flush exit 2024-03-29T11:11:35.200 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.221+0000 7feabcd04640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:11:35.200 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.221+0000 7feabcd04640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:11:35.201 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.222+0000 7feabcd04640 10 monclient: shutdown 2024-03-29T11:11:35.201 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.222+0000 7feabcd04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:35.201 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.222+0000 7feabcd04640 1 -- 172.21.15.195:0/1307115786 >> v1:172.21.15.195:6789/0 conn(0x7feab809ac40 legacy=0x7feab80af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.201 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.222+0000 7feabcd04640 1 -- 172.21.15.195:0/1307115786 shutdown_connections 2024-03-29T11:11:35.201 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.222+0000 7feabcd04640 1 -- 172.21.15.195:0/1307115786 >> 172.21.15.195:0/1307115786 conn(0x7feab8066b00 msgr2=0x7feab80b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.201 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.222+0000 7feabcd04640 1 -- 172.21.15.195:0/1307115786 shutdown_connections 2024-03-29T11:11:35.201 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.222+0000 7feabcd04640 1 -- 172.21.15.195:0/1307115786 wait complete. 2024-03-29T11:11:35.201 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.222+0000 7feabcd04640 1 librados: shutdown 2024-03-29T11:11:35.210 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new ba4bdf82-db18-41a4-b021-a96362bfefb8 4 2024-03-29T11:11:35.259 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:35.281+0000 7f1690c32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:11:35.305 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.327+0000 7fc60e83c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:11:35.330 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:35.352+0000 7f1690c32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:35.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.371+0000 7f9f931a8640 10 monclient: get_monmap_and_config 2024-03-29T11:11:35.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.371+0000 7f9f931a8640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.371+0000 7f9f931a8640 10 monclient: monmap: 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:35.372690+0000 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:35.372690+0000 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:35.350 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.372+0000 7f9f931a8640 1 Processor -- start 2024-03-29T11:11:35.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.372+0000 7f9f931a8640 1 -- start start 2024-03-29T11:11:35.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.372+0000 7f9f931a8640 10 monclient: init 2024-03-29T11:11:35.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.372+0000 7f9f931a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.372+0000 7f9f931a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:35.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.372+0000 7f9f931a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9f8c09ac40 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:35.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9f8c09c870 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:35.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9f8c0b4360 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:35.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c09dec0 con 0x7f9f8c0b4360 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c09da30 con 0x7f9f8c09c870 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c0b39c0 con 0x7f9f8c09ac40 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 10 monclient(hunting): authenticate will time out at 927.644032s 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8bfff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9f8c09c870 0x7f9f8c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36250/0 (socket says 172.21.15.195:36250) 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8bfff640 1 -- 172.21.15.195:0/1302443504 learned_addr learned my addr 172.21.15.195:0/1302443504 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469912380 0 0) 0x7f9f8c09da30 con 0x7f9f8c09c870 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f7c003520 con 0x7f9f8c09c870 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299668004 0 0) 0x7f9f8c0b39c0 con 0x7f9f8c09ac40 2024-03-29T11:11:35.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f8c09da30 con 0x7f9f8c09ac40 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258897441 0 0) 0x7f9f84001490 con 0x7f9f8c0b4360 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f8c0b39c0 con 0x7f9f8c0b4360 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4136757705 0 0) 0x7f9f7c003520 con 0x7f9f8c09c870 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 >> v1:172.21.15.157:6790/0 conn(0x7f9f8c09ac40 legacy=0x7f9f8c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 >> v1:172.21.15.157:6789/0 conn(0x7f9f8c0b4360 legacy=0x7f9f8c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c0b5160 con 0x7f9f8c09c870 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9f78001690 con 0x7f9f8c09c870 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: dump: 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.363 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f78001870 con 0x7f9f8c09c870 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 1 -- 172.21.15.195:0/1302443504 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9f78001c80 con 0x7f9f8c09c870 2024-03-29T11:11:35.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f931a8640 4 set_mon_vals no callback set 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.373+0000 7f9f8b7fe640 10 monclient: dump: 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.374+0000 7f9f931a8640 10 monclient: shutdown 2024-03-29T11:11:35.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.374+0000 7f9f931a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.374+0000 7f9f931a8640 1 -- 172.21.15.195:0/1302443504 >> v1:172.21.15.195:6789/0 conn(0x7f9f8c09c870 legacy=0x7f9f8c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.374+0000 7f9f931a8640 1 -- 172.21.15.195:0/1302443504 shutdown_connections 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.374+0000 7f9f931a8640 1 -- 172.21.15.195:0/1302443504 >> 172.21.15.195:0/1302443504 conn(0x7f9f8c066b00 msgr2=0x7f9f8c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.374+0000 7f9f931a8640 1 -- 172.21.15.195:0/1302443504 shutdown_connections 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.374+0000 7f9f931a8640 1 -- 172.21.15.195:0/1302443504 wait complete. 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.375+0000 7f9f931a8640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.375+0000 7f9f931a8640 10 monclient: monmap: 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.366 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.375+0000 7f9f931a8640 1 librados: starting msgr at 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.375+0000 7f9f931a8640 1 librados: starting objecter 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.375+0000 7f9f931a8640 1 Processor -- start 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.376+0000 7f9f931a8640 1 -- start start 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.376+0000 7f9f931a8640 1 librados: setting wanted keys 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.376+0000 7f9f931a8640 1 librados: calling monclient init 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.376+0000 7f9f931a8640 10 monclient: init 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9f8c09ac40 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9f8c140660 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:35.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9f8c140f80 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c1423a0 con 0x7f9f8c09ac40 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c140140 con 0x7f9f8c140660 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c0b8040 con 0x7f9f8c140f80 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient(hunting): authenticate will time out at 927.648105s 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f8bfff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9f8c140660 0x7f9f8c0b1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36256/0 (socket says 172.21.15.195:36256) 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f8bfff640 1 -- 172.21.15.195:0/1460933449 learned_addr learned my addr 172.21.15.195:0/1460933449 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 client.?.objecter ms_handle_connect 0x7f9f8c140660 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 client.?.objecter ms_handle_connect 0x7f9f8c09ac40 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 client.?.objecter ms_handle_connect 0x7f9f8c140f80 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135021118 0 0) 0x7f9f8c140140 con 0x7f9f8c140660 2024-03-29T11:11:35.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f70004eb0 con 0x7f9f8c140660 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874519707 0 0) 0x7f9f8c1423a0 con 0x7f9f8c09ac40 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f8c140140 con 0x7f9f8c09ac40 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561518549 0 0) 0x7f9f8c0b8040 con 0x7f9f8c140f80 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:11:35.369 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f8c1423a0 con 0x7f9f8c140f80 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2608829020 0 0) 0x7f9f70004eb0 con 0x7f9f8c140660 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f8c0b8040 con 0x7f9f8c140660 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9f78001bc0 con 0x7f9f8c140660 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): dump: 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.370 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3116501371 0 0) 0x7f9f8c1423a0 con 0x7f9f8c140f80 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f70004eb0 con 0x7f9f8c140f80 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 303068731 0 0) 0x7f9f8c140140 con 0x7f9f8c09ac40 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f8c1423a0 con 0x7f9f8c09ac40 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9f84001bc0 con 0x7f9f8c140f80 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): dump: 2024-03-29T11:11:35.371 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9f800018b0 con 0x7f9f8c09ac40 2024-03-29T11:11:35.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient(hunting): dump: 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2771257817 0 0) 0x7f9f8c0b8040 con 0x7f9f8c140660 2024-03-29T11:11:35.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 >> v1:172.21.15.157:6790/0 conn(0x7f9f8c140f80 legacy=0x7f9f8c1e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 >> v1:172.21.15.157:6789/0 conn(0x7f9f8c09ac40 legacy=0x7f9f8c1d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 monclient: found mon.b 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c0b81d0 con 0x7f9f8c140660 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient: _renew_subs 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8c1d7b90 con 0x7f9f8c140660 2024-03-29T11:11:35.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient: _renew_subs 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8c0b19f0 con 0x7f9f8c140660 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.377+0000 7f9f931a8640 1 librados: init done 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f78004bf0 con 0x7f9f8c140660 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9f78004f00 con 0x7f9f8c140660 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 10 monclient: dump: 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.375 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f8affd640 4 set_mon_vals no callback set 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9f780053f0 con 0x7f9f8c140660 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 809770943 0 0) 0x7f9f78006a40 con 0x7f9f8c140660 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 10 client.4138.objecter ms_dispatch 0x7f9f8c000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.378+0000 7f9f897fa640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.379+0000 7f9f931a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.379+0000 7f9f931a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.376 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.379+0000 7f9f931a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.377 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.379+0000 7f9f931a8640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c005080 con 0x7f9f8c140660 2024-03-29T11:11:35.377 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.381+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9f78006640 con 0x7f9f8c140660 2024-03-29T11:11:35.377 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.381+0000 7f9f897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.377 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.381+0000 7f9f897fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:35.377 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:35.399+0000 7f1690c32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:11:35.381 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.402+0000 7fc60e83c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:35.428 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.450+0000 7fc60e83c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:11:35.429 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:35.451+0000 7f1690c32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:11:35.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.453+0000 7f9f931a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ba4bdf82-db18-41a4-b021-a96362bfefb8", "id": 4}] 2024-03-29T11:11:35.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.453+0000 7f9f931a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ba4bdf82-db18-41a4-b021-a96362bfefb8", "id": 4}] 2024-03-29T11:11:35.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.453+0000 7f9f931a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.453+0000 7f9f931a8640 1 -- 172.21.15.195:0/1460933449 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ba4bdf82-db18-41a4-b021-a96362bfefb8", "id": 4} v 0) v1 -- 0x7f9f6c0058b0 con 0x7f9f8c140660 2024-03-29T11:11:35.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.456+0000 7f9f897fa640 1 -- 172.21.15.195:0/1460933449 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ba4bdf82-db18-41a4-b021-a96362bfefb8", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2342504452 0 1697012583) 0x7f9f78005790 con 0x7f9f8c140660 2024-03-29T11:11:35.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.456+0000 7f9f897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ba4bdf82-db18-41a4-b021-a96362bfefb8", "id": 4}] 2024-03-29T11:11:35.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.456+0000 7f9f897fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:35.436 INFO:teuthology.orchestra.run.smithi195.stdout:4 2024-03-29T11:11:35.437 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 10 librados: watch_flush enter 2024-03-29T11:11:35.437 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 10 librados: watch_flush exit 2024-03-29T11:11:35.437 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:11:35.437 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:11:35.437 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 10 monclient: shutdown 2024-03-29T11:11:35.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:35.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 1 -- 172.21.15.195:0/1460933449 >> v1:172.21.15.195:6789/0 conn(0x7f9f8c140660 legacy=0x7f9f8c0b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 1 -- 172.21.15.195:0/1460933449 shutdown_connections 2024-03-29T11:11:35.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 1 -- 172.21.15.195:0/1460933449 >> 172.21.15.195:0/1460933449 conn(0x7f9f8c066b00 msgr2=0x7f9f8c13c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 1 -- 172.21.15.195:0/1460933449 shutdown_connections 2024-03-29T11:11:35.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 1 -- 172.21.15.195:0/1460933449 wait complete. 2024-03-29T11:11:35.438 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.457+0000 7f9f931a8640 1 librados: shutdown 2024-03-29T11:11:35.445 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 527f85da-b570-45f6-bd5f-81434c94f48a 5 2024-03-29T11:11:35.477 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:35.499+0000 7f1690c32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:11:35.484 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.506+0000 7fc60e83c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:11:35.534 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.556+0000 7fc60e83c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:11:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.567+0000 7f9b79ac8640 10 monclient: get_monmap_and_config 2024-03-29T11:11:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.567+0000 7f9b79ac8640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.567+0000 7f9b79ac8640 10 monclient: monmap: 2024-03-29T11:11:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:35.568637+0000 2024-03-29T11:11:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:35.568637+0000 2024-03-29T11:11:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.546 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:35.547 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:35.547 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:35.547 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.568+0000 7f9b79ac8640 1 Processor -- start 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.568+0000 7f9b79ac8640 1 -- start start 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.568+0000 7f9b79ac8640 10 monclient: init 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.568+0000 7f9b79ac8640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.568+0000 7f9b79ac8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.568+0000 7f9b79ac8640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7409ac40 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7409c870 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 10 monclient(hunting): picked mon.noname-a con 0x7f9b740b4360 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7409dec0 con 0x7f9b740b4360 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7409da30 con 0x7f9b7409c870 2024-03-29T11:11:35.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b740b39c0 con 0x7f9b7409ac40 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 10 monclient(hunting): authenticate will time out at 927.840024s 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b72ffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9b7409c870 0x7f9b7413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36258/0 (socket says 172.21.15.195:36258) 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b72ffd640 1 -- 172.21.15.195:0/3788853191 learned_addr learned my addr 172.21.15.195:0/3788853191 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196725334 0 0) 0x7f9b7409da30 con 0x7f9b7409c870 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b5c003520 con 0x7f9b7409c870 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214610824 0 0) 0x7f9b740b39c0 con 0x7f9b7409ac40 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7409da30 con 0x7f9b7409ac40 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771514003 0 0) 0x7f9b6c001490 con 0x7f9b740b4360 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b740b39c0 con 0x7f9b740b4360 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3487486130 0 0) 0x7f9b5c003520 con 0x7f9b7409c870 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 >> v1:172.21.15.157:6790/0 conn(0x7f9b7409ac40 legacy=0x7f9b740b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 >> v1:172.21.15.157:6789/0 conn(0x7f9b740b4360 legacy=0x7f9b7414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 monclient: found mon.noname-c 2024-03-29T11:11:35.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b740b5160 con 0x7f9b7409c870 2024-03-29T11:11:35.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9b64001690 con 0x7f9b7409c870 2024-03-29T11:11:35.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:11:35.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.565 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.565 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:35.565 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: dump: 2024-03-29T11:11:35.565 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.565 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.565 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.565 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.566 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.566 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.566 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.566 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.566 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.566 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.567 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.567 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b64001870 con 0x7f9b7409c870 2024-03-29T11:11:35.567 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.567 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:35.567 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 1 -- 172.21.15.195:0/3788853191 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9b64001c80 con 0x7f9b7409c870 2024-03-29T11:11:35.567 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.567 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b79ac8640 4 set_mon_vals no callback set 2024-03-29T11:11:35.568 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:35.568 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.569+0000 7f9b727fc640 10 monclient: dump: 2024-03-29T11:11:35.568 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.568 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.568 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.568 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.568 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.569 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.569 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.569 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.569 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.569 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.569 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.570+0000 7f9b79ac8640 10 monclient: shutdown 2024-03-29T11:11:35.570 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.570+0000 7f9b79ac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:35.570 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.570+0000 7f9b79ac8640 1 -- 172.21.15.195:0/3788853191 >> v1:172.21.15.195:6789/0 conn(0x7f9b7409c870 legacy=0x7f9b7413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.570 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.570+0000 7f9b79ac8640 1 -- 172.21.15.195:0/3788853191 shutdown_connections 2024-03-29T11:11:35.570 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.570+0000 7f9b79ac8640 1 -- 172.21.15.195:0/3788853191 >> 172.21.15.195:0/3788853191 conn(0x7f9b74066b00 msgr2=0x7f9b74066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.570 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.570+0000 7f9b79ac8640 1 -- 172.21.15.195:0/3788853191 shutdown_connections 2024-03-29T11:11:35.570 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.570+0000 7f9b79ac8640 1 -- 172.21.15.195:0/3788853191 wait complete. 2024-03-29T11:11:35.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.571+0000 7f9b79ac8640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.571+0000 7f9b79ac8640 10 monclient: monmap: 2024-03-29T11:11:35.571 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:35.571 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:35.571 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-29T11:11:35.571 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-29T11:11:35.571 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.572 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.572 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:35.572 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:35.572 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:35.572 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.571+0000 7f9b79ac8640 1 librados: starting msgr at 2024-03-29T11:11:35.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.571+0000 7f9b79ac8640 1 librados: starting objecter 2024-03-29T11:11:35.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.571+0000 7f9b79ac8640 1 Processor -- start 2024-03-29T11:11:35.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.571+0000 7f9b79ac8640 1 -- start start 2024-03-29T11:11:35.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.571+0000 7f9b79ac8640 1 librados: setting wanted keys 2024-03-29T11:11:35.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.571+0000 7f9b79ac8640 1 librados: calling monclient init 2024-03-29T11:11:35.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.571+0000 7f9b79ac8640 10 monclient: init 2024-03-29T11:11:35.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b79ac8640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b79ac8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:11:35.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b79ac8640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7409ac40 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:35.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b79ac8640 10 monclient(hunting): picked mon.noname-a con 0x7f9b74140660 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:35.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b79ac8640 10 monclient(hunting): picked mon.noname-c con 0x7f9b74140f80 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:35.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b79ac8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b741423a0 con 0x7f9b74140660 2024-03-29T11:11:35.575 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b79ac8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b74140140 con 0x7f9b7409ac40 2024-03-29T11:11:35.575 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b79ac8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b740b8040 con 0x7f9b74140f80 2024-03-29T11:11:35.575 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b79ac8640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.575 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b79ac8640 10 monclient(hunting): authenticate will time out at 927.843171s 2024-03-29T11:11:35.575 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b737fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9b7409ac40 0x7f9b741d7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36270/0 (socket says 172.21.15.195:36270) 2024-03-29T11:11:35.575 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b737fe640 1 -- 172.21.15.195:0/4012398832 learned_addr learned my addr 172.21.15.195:0/4012398832 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:35.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 client.?.objecter ms_handle_connect 0x7f9b7409ac40 2024-03-29T11:11:35.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 client.?.objecter ms_handle_connect 0x7f9b74140660 2024-03-29T11:11:35.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 client.?.objecter ms_handle_connect 0x7f9b74140f80 2024-03-29T11:11:35.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171090536 0 0) 0x7f9b74140140 con 0x7f9b7409ac40 2024-03-29T11:11:35.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:11:35.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b60004eb0 con 0x7f9b7409ac40 2024-03-29T11:11:35.578 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653198960 0 0) 0x7f9b741423a0 con 0x7f9b74140660 2024-03-29T11:11:35.578 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.578 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.578 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.578 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:11:35.578 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b74140140 con 0x7f9b74140660 2024-03-29T11:11:35.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812006215 0 0) 0x7f9b740b8040 con 0x7f9b74140f80 2024-03-29T11:11:35.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:11:35.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b741423a0 con 0x7f9b74140f80 2024-03-29T11:11:35.579 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2768402288 0 0) 0x7f9b60004eb0 con 0x7f9b7409ac40 2024-03-29T11:11:35.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b740b8040 con 0x7f9b7409ac40 2024-03-29T11:11:35.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9b68001960 con 0x7f9b7409ac40 2024-03-29T11:11:35.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:35.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): dump: 2024-03-29T11:11:35.580 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.581 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.581 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.581 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.581 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.581 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.581 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.581 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.582 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.582 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.582 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:35.582 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1244568710 0 0) 0x7f9b741423a0 con 0x7f9b74140f80 2024-03-29T11:11:35.582 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b60004eb0 con 0x7f9b74140f80 2024-03-29T11:11:35.582 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1597807125 0 0) 0x7f9b74140140 con 0x7f9b74140660 2024-03-29T11:11:35.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b741423a0 con 0x7f9b74140660 2024-03-29T11:11:35.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9b6c001bc0 con 0x7f9b74140f80 2024-03-29T11:11:35.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:35.583 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): dump: 2024-03-29T11:11:35.583 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.584 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.584 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.584 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.584 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.584 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.584 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.584 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.584 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.584 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9b64001b80 con 0x7f9b74140660 2024-03-29T11:11:35.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:35.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient(hunting): dump: 2024-03-29T11:11:35.585 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.585 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.585 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.585 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.585 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.586 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.586 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.586 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.586 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.586 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2535989264 0 0) 0x7f9b740b8040 con 0x7f9b7409ac40 2024-03-29T11:11:35.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 >> v1:172.21.15.157:6790/0 conn(0x7f9b74140f80 legacy=0x7f9b741e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 >> v1:172.21.15.157:6789/0 conn(0x7f9b74140660 legacy=0x7f9b740b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 monclient: found mon.b 2024-03-29T11:11:35.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b740b81d0 con 0x7f9b7409ac40 2024-03-29T11:11:35.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.572+0000 7f9b5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.588 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:11:35.588 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 10 monclient: _renew_subs 2024-03-29T11:11:35.588 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.588 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b741d7b90 con 0x7f9b7409ac40 2024-03-29T11:11:35.588 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:35.588 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 10 monclient: _renew_subs 2024-03-29T11:11:35.588 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.588 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b740b19f0 con 0x7f9b7409ac40 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 1 librados: init done 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b68003610 con 0x7f9b7409ac40 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9b68003960 con 0x7f9b7409ac40 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 10 monclient: dump: 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.589 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b71ffb640 4 set_mon_vals no callback set 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9b68003e50 con 0x7f9b7409ac40 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1492717481 0 0) 0x7f9b680057c0 con 0x7f9b7409ac40 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 10 client.4144.objecter ms_dispatch 0x7f9b74000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b5bfff640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.573+0000 7f9b79ac8640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b54005080 con 0x7f9b7409ac40 2024-03-29T11:11:35.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.575+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9b68018dc0 con 0x7f9b7409ac40 2024-03-29T11:11:35.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.575+0000 7f9b5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.575+0000 7f9b5bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:35.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.648+0000 7f9b79ac8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "527f85da-b570-45f6-bd5f-81434c94f48a", "id": 5}] 2024-03-29T11:11:35.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.648+0000 7f9b79ac8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "527f85da-b570-45f6-bd5f-81434c94f48a", "id": 5}] 2024-03-29T11:11:35.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.648+0000 7f9b79ac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.648+0000 7f9b79ac8640 1 -- 172.21.15.195:0/4012398832 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "527f85da-b570-45f6-bd5f-81434c94f48a", "id": 5} v 0) v1 -- 0x7f9b540058b0 con 0x7f9b7409ac40 2024-03-29T11:11:35.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b5bfff640 1 -- 172.21.15.195:0/4012398832 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "527f85da-b570-45f6-bd5f-81434c94f48a", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3531995337 0 1988412176) 0x7f9b68007020 con 0x7f9b7409ac40 2024-03-29T11:11:35.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "527f85da-b570-45f6-bd5f-81434c94f48a", "id": 5}] 2024-03-29T11:11:35.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b5bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:35.630 INFO:teuthology.orchestra.run.smithi195.stdout:5 2024-03-29T11:11:35.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b79ac8640 10 librados: watch_flush enter 2024-03-29T11:11:35.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b79ac8640 10 librados: watch_flush exit 2024-03-29T11:11:35.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b79ac8640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:11:35.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b79ac8640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:11:35.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b79ac8640 10 monclient: shutdown 2024-03-29T11:11:35.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b79ac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:35.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b79ac8640 1 -- 172.21.15.195:0/4012398832 >> v1:172.21.15.195:6789/0 conn(0x7f9b7409ac40 legacy=0x7f9b741d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.651+0000 7f9b79ac8640 1 -- 172.21.15.195:0/4012398832 shutdown_connections 2024-03-29T11:11:35.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.652+0000 7f9b79ac8640 1 -- 172.21.15.195:0/4012398832 >> 172.21.15.195:0/4012398832 conn(0x7f9b74066b00 msgr2=0x7f9b7413c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.652+0000 7f9b79ac8640 1 -- 172.21.15.195:0/4012398832 shutdown_connections 2024-03-29T11:11:35.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.652+0000 7f9b79ac8640 1 -- 172.21.15.195:0/4012398832 wait complete. 2024-03-29T11:11:35.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.652+0000 7f9b79ac8640 1 librados: shutdown 2024-03-29T11:11:35.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 6eeab36f-56c0-4923-bf31-ac9b0c845f6e 6 2024-03-29T11:11:35.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.805+0000 7f7994754640 10 monclient: get_monmap_and_config 2024-03-29T11:11:35.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.805+0000 7f7994754640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.805+0000 7f7994754640 10 monclient: monmap: 2024-03-29T11:11:35.784 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:35.784 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:35.806873+0000 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:35.806873+0000 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.806+0000 7f7994754640 1 Processor -- start 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.806+0000 7f7994754640 1 -- start start 2024-03-29T11:11:35.785 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.806+0000 7f7994754640 10 monclient: init 2024-03-29T11:11:35.795 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7994754640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.795 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7994754640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:35.795 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7994754640 10 monclient(hunting): picked mon.noname-b con 0x7f798c09ac40 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:35.795 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7994754640 10 monclient(hunting): picked mon.noname-a con 0x7f798c09c870 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7994754640 10 monclient(hunting): picked mon.noname-c con 0x7f798c0b4360 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7994754640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c09dec0 con 0x7f798c09c870 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7994754640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c09da30 con 0x7f798c0b4360 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7994754640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0b39c0 con 0x7f798c09ac40 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7994754640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7994754640 10 monclient(hunting): authenticate will time out at 928.078265s 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7992cf1640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f798c0b4360 0x7f798c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36280/0 (socket says 172.21.15.195:36280) 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f7992cf1640 1 -- 172.21.15.195:0/67150566 learned_addr learned my addr 172.21.15.195:0/67150566 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489572440 0 0) 0x7f798c09da30 con 0x7f798c0b4360 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7978003520 con 0x7f798c0b4360 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578756854 0 0) 0x7f798c0b39c0 con 0x7f798c09ac40 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f798c09da30 con 0x7f798c09ac40 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566551750 0 0) 0x7f7988001110 con 0x7f798c09c870 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f798c0b39c0 con 0x7f798c09c870 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 636293993 0 0) 0x7f7978003520 con 0x7f798c0b4360 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 >> v1:172.21.15.157:6790/0 conn(0x7f798c09ac40 legacy=0x7f798c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 >> v1:172.21.15.157:6789/0 conn(0x7f798c09c870 legacy=0x7f798c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 monclient: found mon.noname-c 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c0b5160 con 0x7f798c0b4360 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.807+0000 7f79914ee640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f7994754640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f7994754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7984001a10 con 0x7f798c0b4360 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 10 monclient: dump: 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.799 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f7994754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7984001c60 con 0x7f798c0b4360 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f7994754640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 1 -- 172.21.15.195:0/67150566 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7984001ff0 con 0x7f798c0b4360 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f7994754640 4 set_mon_vals no callback set 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f79914ee640 10 monclient: dump: 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.800 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f7994754640 10 monclient: shutdown 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f7994754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f7994754640 1 -- 172.21.15.195:0/67150566 >> v1:172.21.15.195:6789/0 conn(0x7f798c0b4360 legacy=0x7f798c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.801 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f7994754640 1 -- 172.21.15.195:0/67150566 shutdown_connections 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.808+0000 7f7994754640 1 -- 172.21.15.195:0/67150566 >> 172.21.15.195:0/67150566 conn(0x7f798c066b00 msgr2=0x7f798c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.809+0000 7f7994754640 1 -- 172.21.15.195:0/67150566 shutdown_connections 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.809+0000 7f7994754640 1 -- 172.21.15.195:0/67150566 wait complete. 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.809+0000 7f7994754640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.809+0000 7f7994754640 10 monclient: monmap: 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.802 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.803 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:35.803 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:35.803 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:35.803 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.809+0000 7f7994754640 1 librados: starting msgr at 2024-03-29T11:11:35.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.809+0000 7f7994754640 1 librados: starting objecter 2024-03-29T11:11:35.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 1 Processor -- start 2024-03-29T11:11:35.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 1 -- start start 2024-03-29T11:11:35.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 1 librados: setting wanted keys 2024-03-29T11:11:35.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 1 librados: calling monclient init 2024-03-29T11:11:35.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 10 monclient: init 2024-03-29T11:11:35.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:35.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 10 monclient(hunting): picked mon.noname-c con 0x7f798c09ac40 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:35.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 10 monclient(hunting): picked mon.noname-b con 0x7f798c0ad8f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:35.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 10 monclient(hunting): picked mon.noname-a con 0x7f798c0b8160 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:35.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0afc70 con 0x7f798c0b8160 2024-03-29T11:11:35.804 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0aff50 con 0x7f798c0ad8f0 2024-03-29T11:11:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0ae400 con 0x7f798c09ac40 2024-03-29T11:11:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7994754640 10 monclient(hunting): authenticate will time out at 928.081626s 2024-03-29T11:11:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7991cef640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f798c0ad8f0 0x7f798c1db200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36282/0 (socket says 172.21.15.195:36282) 2024-03-29T11:11:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7992cf1640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f798c0b8160 0x7f798c1e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.195:60942/0 (socket says 172.21.15.195:60942) 2024-03-29T11:11:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7991cef640 1 -- 172.21.15.195:0/2748740923 learned_addr learned my addr 172.21.15.195:0/2748740923 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7982ffd640 10 client.?.objecter ms_handle_connect 0x7f798c0ad8f0 2024-03-29T11:11:35.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7982ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7982ffd640 10 client.?.objecter ms_handle_connect 0x7f798c0b8160 2024-03-29T11:11:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7982ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7982ffd640 10 client.?.objecter ms_handle_connect 0x7f798c09ac40 2024-03-29T11:11:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.810+0000 7f7982ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683513222 0 0) 0x7f798c0aff50 con 0x7f798c0ad8f0 2024-03-29T11:11:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.806 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:11:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7970004eb0 con 0x7f798c0ad8f0 2024-03-29T11:11:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340180845 0 0) 0x7f798c0afc70 con 0x7f798c0b8160 2024-03-29T11:11:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:11:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f798c0aff50 con 0x7f798c0b8160 2024-03-29T11:11:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094154325 0 0) 0x7f798c0ae400 con 0x7f798c09ac40 2024-03-29T11:11:35.807 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f798c0afc70 con 0x7f798c09ac40 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 779230450 0 0) 0x7f7970004eb0 con 0x7f798c0ad8f0 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f798c0ae400 con 0x7f798c0ad8f0 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f79880017e0 con 0x7f798c0ad8f0 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): dump: 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.808 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2888983221 0 0) 0x7f798c0afc70 con 0x7f798c09ac40 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7970004eb0 con 0x7f798c09ac40 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4261334353 0 0) 0x7f798c0aff50 con 0x7f798c0b8160 2024-03-29T11:11:35.809 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f798c0afc70 con 0x7f798c0b8160 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f797c0018b0 con 0x7f798c09ac40 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): dump: 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.810 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7984001dd0 con 0x7f798c0b8160 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient(hunting): dump: 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.811 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1639765919 0 0) 0x7f798c0ae400 con 0x7f798c0ad8f0 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 >> v1:172.21.15.157:6790/0 conn(0x7f798c09ac40 legacy=0x7f798c1daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 >> v1:172.21.15.157:6789/0 conn(0x7f798c0b8160 legacy=0x7f798c1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 monclient: found mon.b 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c1e6510 con 0x7f798c0ad8f0 2024-03-29T11:11:35.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7994754640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:11:35.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7994754640 10 monclient: _renew_subs 2024-03-29T11:11:35.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7994754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7994754640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798c1e34e0 con 0x7f798c0ad8f0 2024-03-29T11:11:35.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7994754640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:35.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7994754640 10 monclient: _renew_subs 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7994754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7994754640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f798c1e3970 con 0x7f798c0ad8f0 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7994754640 1 librados: init done 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7988001d50 con 0x7f798c0ad8f0 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f79880037b0 con 0x7f798c0ad8f0 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7990ced640 4 set_mon_vals no callback set 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 monclient: dump: 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.814 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7988003d20 con 0x7f798c0ad8f0 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4163710457 0 0) 0x7f79880067a0 con 0x7f798c0ad8f0 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 10 client.4150.objecter ms_dispatch 0x7f798c000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:11:35.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:11:35.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:11:35.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.811+0000 7f7982ffd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T11:11:35.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.812+0000 7f7994754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.812+0000 7f7994754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.812+0000 7f7994754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.812+0000 7f7994754640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7974005080 con 0x7f798c0ad8f0 2024-03-29T11:11:35.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.814+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7988004260 con 0x7f798c0ad8f0 2024-03-29T11:11:35.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.814+0000 7f7982ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:35.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.814+0000 7f7982ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:35.858 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:35.880+0000 7f1690c32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:11:35.864 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.886+0000 7f7994754640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6eeab36f-56c0-4923-bf31-ac9b0c845f6e", "id": 6}] 2024-03-29T11:11:35.864 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.886+0000 7f7994754640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6eeab36f-56c0-4923-bf31-ac9b0c845f6e", "id": 6}] 2024-03-29T11:11:35.864 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.886+0000 7f7994754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.864 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.886+0000 7f7994754640 1 -- 172.21.15.195:0/2748740923 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6eeab36f-56c0-4923-bf31-ac9b0c845f6e", "id": 6} v 0) v1 -- 0x7f79740058b0 con 0x7f798c0ad8f0 2024-03-29T11:11:35.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.889+0000 7f7982ffd640 1 -- 172.21.15.195:0/2748740923 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6eeab36f-56c0-4923-bf31-ac9b0c845f6e", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3952024433 0 1113811849) 0x7f7988006a30 con 0x7f798c0ad8f0 2024-03-29T11:11:35.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.889+0000 7f7982ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6eeab36f-56c0-4923-bf31-ac9b0c845f6e", "id": 6}] 2024-03-29T11:11:35.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.889+0000 7f7982ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:35.868 INFO:teuthology.orchestra.run.smithi195.stdout:6 2024-03-29T11:11:35.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 10 librados: watch_flush enter 2024-03-29T11:11:35.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 10 librados: watch_flush exit 2024-03-29T11:11:35.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T11:11:35.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 10 client.4150.objecter successfully canceled tick 2024-03-29T11:11:35.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 10 monclient: shutdown 2024-03-29T11:11:35.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:35.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 1 -- 172.21.15.195:0/2748740923 >> v1:172.21.15.195:6789/0 conn(0x7f798c0ad8f0 legacy=0x7f798c1db200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 1 -- 172.21.15.195:0/2748740923 shutdown_connections 2024-03-29T11:11:35.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 1 -- 172.21.15.195:0/2748740923 >> 172.21.15.195:0/2748740923 conn(0x7f798c066b00 msgr2=0x7f798c13a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:35.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 1 -- 172.21.15.195:0/2748740923 shutdown_connections 2024-03-29T11:11:35.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 1 -- 172.21.15.195:0/2748740923 wait complete. 2024-03-29T11:11:35.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:35.890+0000 7f7994754640 1 librados: shutdown 2024-03-29T11:11:35.879 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 722f1ec5-66b1-4d7c-993d-25ec8b415f2a 7 2024-03-29T11:11:35.935 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:35.957+0000 7fc60e83c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:11:35.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.001+0000 7f73f8b59640 10 monclient: get_monmap_and_config 2024-03-29T11:11:35.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.001+0000 7f73f8b59640 10 monclient: build_initial_monmap 2024-03-29T11:11:35.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.001+0000 7f73f8b59640 10 monclient: monmap: 2024-03-29T11:11:35.980 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:35.980 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.980 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:36.002469+0000 2024-03-29T11:11:35.980 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:36.002469+0000 2024-03-29T11:11:35.980 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:35.980 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.980 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:35.980 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:35.981 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:35.981 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.001+0000 7f73f8b59640 1 Processor -- start 2024-03-29T11:11:35.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 1 -- start start 2024-03-29T11:11:35.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 10 monclient: init 2024-03-29T11:11:35.993 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:35.993 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:11:35.993 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 10 monclient(hunting): picked mon.noname-a con 0x7f73f409ac40 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:35.993 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 10 monclient(hunting): picked mon.noname-c con 0x7f73f409c870 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:35.993 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 10 monclient(hunting): picked mon.noname-b con 0x7f73f40b4360 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:35.993 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f409dec0 con 0x7f73f409ac40 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f409da30 con 0x7f73f409c870 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f40b39c0 con 0x7f73f40b4360 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f8b59640 10 monclient(hunting): authenticate will time out at 928.273749s 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f1d9b640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f73f409c870 0x7f73f413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36288/0 (socket says 172.21.15.195:36288) 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.002+0000 7f73f1d9b640 1 -- 172.21.15.195:0/364131718 learned_addr learned my addr 172.21.15.195:0/364131718 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063725556 0 0) 0x7f73f409da30 con 0x7f73f409c870 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73dc003520 con 0x7f73f409c870 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607883537 0 0) 0x7f73f40b39c0 con 0x7f73f40b4360 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f409da30 con 0x7f73f40b4360 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150097939 0 0) 0x7f73e4001110 con 0x7f73f409ac40 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:35.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f40b39c0 con 0x7f73f409ac40 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 281185191 0 0) 0x7f73dc003520 con 0x7f73f409c870 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 >> v1:172.21.15.157:6790/0 conn(0x7f73f40b4360 legacy=0x7f73f414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 >> v1:172.21.15.157:6789/0 conn(0x7f73f409ac40 legacy=0x7f73f40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: _finish_hunting 0 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 monclient: found mon.noname-c 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f40b5160 con 0x7f73f409c870 2024-03-29T11:11:35.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: _finish_auth 0 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: _check_auth_tickets 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f8b59640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73e8001690 con 0x7f73f409c870 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f8b59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: dump: 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.997 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f8b59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73e8001870 con 0x7f73f409c870 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f8b59640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:35.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 1 -- 172.21.15.195:0/364131718 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73e8001c80 con 0x7f73f409c870 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f8b59640 4 set_mon_vals no callback set 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.003+0000 7f73f159a640 10 monclient: dump: 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:35.999 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.004+0000 7f73f8b59640 10 monclient: shutdown 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.004+0000 7f73f8b59640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.004+0000 7f73f8b59640 1 -- 172.21.15.195:0/364131718 >> v1:172.21.15.195:6789/0 conn(0x7f73f409c870 legacy=0x7f73f413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.004+0000 7f73f8b59640 1 -- 172.21.15.195:0/364131718 shutdown_connections 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.004+0000 7f73f8b59640 1 -- 172.21.15.195:0/364131718 >> 172.21.15.195:0/364131718 conn(0x7f73f4066b00 msgr2=0x7f73f4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.004+0000 7f73f8b59640 1 -- 172.21.15.195:0/364131718 shutdown_connections 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.004+0000 7f73f8b59640 1 -- 172.21.15.195:0/364131718 wait complete. 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.004+0000 7f73f8b59640 10 monclient: build_initial_monmap 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 10 monclient: monmap: 2024-03-29T11:11:36.000 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 1 librados: starting msgr at 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 1 librados: starting objecter 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 1 Processor -- start 2024-03-29T11:11:36.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 1 -- start start 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 1 librados: setting wanted keys 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 1 librados: calling monclient init 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 10 monclient: init 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 10 monclient(hunting): picked mon.noname-a con 0x7f73f409ac40 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 10 monclient(hunting): picked mon.noname-c con 0x7f73f40b07a0 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 10 monclient(hunting): picked mon.noname-b con 0x7f73f41d7cd0 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f40acd70 con 0x7f73f409ac40 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f41d8100 con 0x7f73f41d7cd0 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f411f8d0 con 0x7f73f40b07a0 2024-03-29T11:11:36.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.005+0000 7f73f8b59640 10 monclient(hunting): authenticate will time out at 928.276915s 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f2d9d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f73f41d7cd0 0x7f73f41e1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:36294/0 (socket says 172.21.15.195:36294) 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f2d9d640 1 -- 172.21.15.195:0/596896936 learned_addr learned my addr 172.21.15.195:0/596896936 (peer_addr_for_me v1:172.21.15.195:0/0) 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f259c640 1 --1- 172.21.15.195:0/596896936 >> v1:172.21.15.157:6789/0 conn(0x7f73f409ac40 0x7f73f40afa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.195:60970/0 (socket says 172.21.15.195:60970) 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 client.?.objecter ms_handle_connect 0x7f73f41d7cd0 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 client.?.objecter ms_handle_connect 0x7f73f409ac40 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 client.?.objecter ms_handle_connect 0x7f73f40b07a0 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699933904 0 0) 0x7f73f41d8100 con 0x7f73f41d7cd0 2024-03-29T11:11:36.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d4004eb0 con 0x7f73f41d7cd0 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379013106 0 0) 0x7f73f40acd70 con 0x7f73f409ac40 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): my global_id is 4161 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f41d8100 con 0x7f73f409ac40 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801776034 0 0) 0x7f73f411f8d0 con 0x7f73f40b07a0 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:36.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): my global_id is 4160 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f40acd70 con 0x7f73f40b07a0 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3731912416 0 0) 0x7f73d4004eb0 con 0x7f73f41d7cd0 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73f411f8d0 con 0x7f73f41d7cd0 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73ec001bb0 con 0x7f73f41d7cd0 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.005 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4127762262 0 0) 0x7f73f41d8100 con 0x7f73f409ac40 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73d4004eb0 con 0x7f73f409ac40 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73e40018e0 con 0x7f73f409ac40 2024-03-29T11:11:36.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:36.007 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2867639336 0 0) 0x7f73f40acd70 con 0x7f73f40b07a0 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73f41d8100 con 0x7f73f40b07a0 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73e8001b80 con 0x7f73f40b07a0 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.008 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1079397208 0 0) 0x7f73f411f8d0 con 0x7f73f41d7cd0 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 >> v1:172.21.15.157:6790/0 conn(0x7f73f40b07a0 legacy=0x7f73f41d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 >> v1:172.21.15.157:6789/0 conn(0x7f73f409ac40 legacy=0x7f73f40afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 monclient: found mon.b 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:36.009 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f41e64c0 con 0x7f73f41d7cd0 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f8b59640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f8b59640 10 monclient: _renew_subs 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f8b59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f8b59640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f41e3490 con 0x7f73f41d7cd0 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f8b59640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f8b59640 10 monclient: _renew_subs 2024-03-29T11:11:36.010 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f8b59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f8b59640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f41e39c0 con 0x7f73f41d7cd0 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f8b59640 1 librados: init done 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73ec0020a0 con 0x7f73f41d7cd0 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73ec003b30 con 0x7f73f41d7cd0 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 10 monclient: dump: 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.011 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73f0d99640 4 set_mon_vals no callback set 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.006+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f73ec004020 con 0x7f73f41d7cd0 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.007+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 938578936 0 0) 0x7f73ec005c90 con 0x7f73f41d7cd0 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.007+0000 7f73e2ffd640 10 client.4156.objecter ms_dispatch 0x7f73f4000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.007+0000 7f73e2ffd640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.007+0000 7f73e2ffd640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:11:36.012 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.007+0000 7f73e2ffd640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-29T11:11:36.013 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.007+0000 7f73f8b59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:36.013 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.007+0000 7f73f8b59640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:36.013 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.007+0000 7f73f8b59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:36.013 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.007+0000 7f73f8b59640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d8005080 con 0x7f73f41d7cd0 2024-03-29T11:11:36.013 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.009+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f73ec0042d0 con 0x7f73f41d7cd0 2024-03-29T11:11:36.013 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.009+0000 7f73e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:36.013 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.009+0000 7f73e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:36.013 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:36.018+0000 7f1690c32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:11:36.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.081+0000 7f73f8b59640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "722f1ec5-66b1-4d7c-993d-25ec8b415f2a", "id": 7}] 2024-03-29T11:11:36.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.081+0000 7f73f8b59640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "722f1ec5-66b1-4d7c-993d-25ec8b415f2a", "id": 7}] 2024-03-29T11:11:36.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.081+0000 7f73f8b59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:36.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.081+0000 7f73f8b59640 1 -- 172.21.15.195:0/596896936 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd new", "uuid": "722f1ec5-66b1-4d7c-993d-25ec8b415f2a", "id": 7} v 0) v1 -- 0x7f73d80058b0 con 0x7f73f41d7cd0 2024-03-29T11:11:36.062 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.084+0000 7f73e2ffd640 1 -- 172.21.15.195:0/596896936 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "722f1ec5-66b1-4d7c-993d-25ec8b415f2a", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1732841068 0 1371665406) 0x7f73ec004470 con 0x7f73f41d7cd0 2024-03-29T11:11:36.062 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.084+0000 7f73e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "722f1ec5-66b1-4d7c-993d-25ec8b415f2a", "id": 7}] 2024-03-29T11:11:36.063 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.084+0000 7f73e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:36.063 INFO:teuthology.orchestra.run.smithi195.stdout:7 2024-03-29T11:11:36.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 10 librados: watch_flush enter 2024-03-29T11:11:36.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 10 librados: watch_flush exit 2024-03-29T11:11:36.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-29T11:11:36.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 10 client.4156.objecter successfully canceled tick 2024-03-29T11:11:36.065 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 10 monclient: shutdown 2024-03-29T11:11:36.065 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:36.065 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 1 -- 172.21.15.195:0/596896936 >> v1:172.21.15.195:6789/0 conn(0x7f73f41d7cd0 legacy=0x7f73f41e1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.065 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 1 -- 172.21.15.195:0/596896936 shutdown_connections 2024-03-29T11:11:36.065 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 1 -- 172.21.15.195:0/596896936 >> 172.21.15.195:0/596896936 conn(0x7f73f4066b00 msgr2=0x7f73f40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:36.065 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 1 -- 172.21.15.195:0/596896936 shutdown_connections 2024-03-29T11:11:36.065 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 1 -- 172.21.15.195:0/596896936 wait complete. 2024-03-29T11:11:36.065 INFO:teuthology.orchestra.run.smithi195.stderr:2024-03-29T11:11:36.085+0000 7f73f8b59640 1 librados: shutdown 2024-03-29T11:11:36.074 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:11:36.074 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:11:36.076 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:36.098+0000 7fc60e83c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:11:36.077 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:11:36.077 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:11:36.077 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:11:36.079 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:11:36.079 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:11:36.079 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:11:36.082 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:11:36.082 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:11:36.082 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:11:36.084 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:11:36.084 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:11:36.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:11:36.086 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:11:36.087 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:11:36.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:11:36.089 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:11:36.089 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:11:36.089 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:11:36.091 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:11:36.091 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:11:36.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:11:36.093 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:11:36.093 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:36.144 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:36.166+0000 7f1690c32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:11:36.197 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:36.219+0000 7f1690c32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:11:36.213 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.235+0000 7f60ca6cf640 10 monclient: get_monmap_and_config 2024-03-29T11:11:36.213 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.235+0000 7f60ca6cf640 10 monclient: build_initial_monmap 2024-03-29T11:11:36.213 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.235+0000 7f60ca6cf640 10 monclient: monmap: 2024-03-29T11:11:36.213 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-29T11:11:36.213 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.213 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:36.236134+0000 2024-03-29T11:11:36.213 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:36.236134+0000 2024-03-29T11:11:36.213 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:36.213 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:36.214 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:36.214 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:36.214 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:36.214 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:36.223 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.235+0000 7f60ca6cf640 1 Processor -- start 2024-03-29T11:11:36.223 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.235+0000 7f60ca6cf640 1 -- start start 2024-03-29T11:11:36.223 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.235+0000 7f60ca6cf640 10 monclient: init 2024-03-29T11:11:36.223 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60ca6cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60ca6cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60ca6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f60c406e500 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60ca6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f60c406f7d0 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60ca6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f60c413b920 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60ca6cf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c413bd70 con 0x7f60c406f7d0 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60ca6cf640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c4070ec0 con 0x7f60c413b920 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60ca6cf640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c4070aa0 con 0x7f60c406e500 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60ca6cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60ca6cf640 10 monclient(hunting): authenticate will time out at 908.433280s 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c37fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f60c406f7d0 0x7f60c41418e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:40670/0 (socket says 172.21.15.157:40670) 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c37fe640 1 -- 172.21.15.157:0/1073016000 learned_addr learned my addr 172.21.15.157:0/1073016000 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833789350 0 0) 0x7f60c413bd70 con 0x7f60c406f7d0 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:36.224 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 10 monclient(hunting): my global_id is 4182 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60ac004630 con 0x7f60c406f7d0 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519221383 0 0) 0x7f60c4070aa0 con 0x7f60c406e500 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.236+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c413bd70 con 0x7f60c406e500 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550087828 0 0) 0x7f60c4070ec0 con 0x7f60c413b920 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:36.225 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c4070aa0 con 0x7f60c413b920 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2144828262 0 0) 0x7f60ac004630 con 0x7f60c406f7d0 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 >> v1:172.21.15.157:6790/0 conn(0x7f60c406e500 legacy=0x7f60c413b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 >> v1:172.21.15.195:6789/0 conn(0x7f60c413b920 legacy=0x7f60c4149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 1 monclient: found mon.noname-a 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c4072aa0 con 0x7f60c406f7d0 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f60b4004cb0 con 0x7f60c406f7d0 2024-03-29T11:11:36.226 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: dump: 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60ca6cf640 5 monclient: authenticate success, global_id 4182 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60ca6cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:36.227 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60b4004e90 con 0x7f60c406f7d0 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 1 -- 172.21.15.157:0/1073016000 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f60b4001320 con 0x7f60c406f7d0 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60c2ffd640 10 monclient: dump: 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.228 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60ca6cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60ca6cf640 4 set_mon_vals no callback set 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60ca6cf640 10 monclient: shutdown 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60ca6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60ca6cf640 1 -- 172.21.15.157:0/1073016000 >> v1:172.21.15.157:6789/0 conn(0x7f60c406f7d0 legacy=0x7f60c41418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60ca6cf640 1 -- 172.21.15.157:0/1073016000 shutdown_connections 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.237+0000 7f60ca6cf640 1 -- 172.21.15.157:0/1073016000 >> 172.21.15.157:0/1073016000 conn(0x7f60c40b3cc0 msgr2=0x7f60c40b40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.238+0000 7f60ca6cf640 1 -- 172.21.15.157:0/1073016000 shutdown_connections 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.238+0000 7f60ca6cf640 1 -- 172.21.15.157:0/1073016000 wait complete. 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.238+0000 7f60ca6cf640 10 monclient: build_initial_monmap 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.238+0000 7f60ca6cf640 10 monclient: monmap: 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 0 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:36.229 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 0.000000 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:created 0.000000 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.238+0000 7f60ca6cf640 1 librados: starting msgr at 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.238+0000 7f60ca6cf640 1 librados: starting objecter 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.238+0000 7f60ca6cf640 1 Processor -- start 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 1 -- start start 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 1 librados: setting wanted keys 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 1 librados: calling monclient init 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 10 monclient: init 2024-03-29T11:11:36.230 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f60c406e500 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f60c406f7d0 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60c3fff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f60c406e500 0x7f60c4133d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:59580/0 (socket says 172.21.15.157:59580) 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f60c413b920 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60c3fff640 1 -- 172.21.15.157:0/2338109834 learned_addr learned my addr 172.21.15.157:0/2338109834 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c413e240 con 0x7f60c413b920 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c413e6e0 con 0x7f60c406f7d0 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60c0ff9640 10 client.?.objecter ms_handle_connect 0x7f60c406e500 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60c0ff9640 10 client.?.objecter ms_handle_connect 0x7f60c413b920 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c413e870 con 0x7f60c406e500 2024-03-29T11:11:36.231 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.239+0000 7f60ca6cf640 10 monclient(hunting): authenticate will time out at 908.436842s 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 client.?.objecter ms_handle_connect 0x7f60c406f7d0 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253135941 0 0) 0x7f60c413e240 con 0x7f60c413b920 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60b0004eb0 con 0x7f60c413b920 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235206551 0 0) 0x7f60c413e6e0 con 0x7f60c406f7d0 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:11:36.232 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c413e240 con 0x7f60c406f7d0 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396079266 0 0) 0x7f60c413e870 con 0x7f60c406e500 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c413e6e0 con 0x7f60c406e500 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2962100050 0 0) 0x7f60c413e240 con 0x7f60c406f7d0 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60c413e870 con 0x7f60c406f7d0 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f60b40013d0 con 0x7f60c406f7d0 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.233 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1136113126 0 0) 0x7f60c413e6e0 con 0x7f60c406e500 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60c413e240 con 0x7f60c406e500 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f60b80012d0 con 0x7f60c406e500 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:36.234 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 559478819 0 0) 0x7f60b0004eb0 con 0x7f60c413b920 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60c413e6e0 con 0x7f60c413b920 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f60bc001b50 con 0x7f60c413b920 2024-03-29T11:11:36.235 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3095477358 0 0) 0x7f60c413e870 con 0x7f60c406f7d0 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 >> v1:172.21.15.157:6790/0 conn(0x7f60c406e500 legacy=0x7f60c4133d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.236 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 >> v1:172.21.15.157:6789/0 conn(0x7f60c413b920 legacy=0x7f60c41d6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 monclient: found mon.b 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c413ea00 con 0x7f60c406f7d0 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60ca6cf640 5 monclient: authenticate success, global_id 4186 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60ca6cf640 10 monclient: _renew_subs 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60ca6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c40b29f0 con 0x7f60c406f7d0 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60b4001890 con 0x7f60c406f7d0 2024-03-29T11:11:36.237 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f60b4001c20 con 0x7f60c406f7d0 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 monclient: dump: 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:epoch 1 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:36.238 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c27fc640 4 set_mon_vals no callback set 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f60b4005390 con 0x7f60c406f7d0 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60ca6cf640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60ca6cf640 10 monclient: _renew_subs 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60ca6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c4134480 con 0x7f60c406f7d0 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60ca6cf640 1 librados: init done 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f60b4005e50 con 0x7f60c406f7d0 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 10 client.4186.objecter ms_dispatch 0x7f60c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.240+0000 7f60c0ff9640 20 client.4186.objecter dump_active .. 0 homeless 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.241+0000 7f60ca6cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.241+0000 7f60ca6cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:36.239 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.241+0000 7f60ca6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:36.240 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.241+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c404f730 con 0x7f60c406f7d0 2024-03-29T11:11:36.240 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.244+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f60b4005780 con 0x7f60c406f7d0 2024-03-29T11:11:36.240 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.244+0000 7f60c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:36.240 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.244+0000 7f60c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:36.240 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:36.246+0000 7fc60e83c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:11:36.270 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:36.292+0000 7fc60e83c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:11:36.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.304+0000 7f60ca6cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:36.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.304+0000 7f60ca6cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:36.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.304+0000 7f60ca6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:36.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.304+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60c4138ff0 con 0x7f60c406f7d0 2024-03-29T11:11:36.283 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.304+0000 7f60c0ff9640 1 -- 172.21.15.157:0/2338109834 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f60b4005920 con 0x7f60c406f7d0 2024-03-29T11:11:36.283 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.304+0000 7f60c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:36.283 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.304+0000 7f60c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:36.283 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-29T11:11:36.283 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:36.284 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 10 librados: watch_flush enter 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 10 librados: watch_flush exit 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 20 client.4186.objecter shutdown clearing up homeless session... 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 10 client.4186.objecter successfully canceled tick 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 10 monclient: shutdown 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 >> v1:172.21.15.195:6789/0 conn(0x7f60c406f7d0 legacy=0x7f60c41d65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 shutdown_connections 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 >> 172.21.15.157:0/2338109834 conn(0x7f60c40b3cc0 msgr2=0x7f60c40b40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 shutdown_connections 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 1 -- 172.21.15.157:0/2338109834 wait complete. 2024-03-29T11:11:36.285 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-29T11:11:36.305+0000 7f60ca6cf640 1 librados: shutdown 2024-03-29T11:11:36.294 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:11:36.294 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:11:36.314 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:36.336+0000 7f1690c32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:36.382 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:36.404+0000 7fc60e83c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:36.408 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:36.430+0000 7f1690c32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:11:36.478 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:36.500+0000 7fc60e83c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:11:36.563 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:36.584+0000 7f1690c32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:36.605 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:36.627+0000 7f1690c32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:36.632 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:36.653+0000 7fc60e83c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:11:36.674 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:36.695+0000 7fc60e83c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:11:36.691 INFO:tasks.ceph.osd.2.smithi157.stderr:2024-03-29T11:11:36.713+0000 7f7505a66640 -1 Falling back to public interface 2024-03-29T11:11:36.692 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-03-29T11:11:36.714+0000 7fcbe7613640 -1 Falling back to public interface 2024-03-29T11:11:36.694 INFO:tasks.ceph.osd.4.smithi195.stderr:2024-03-29T11:11:36.716+0000 7f5043c11640 -1 Falling back to public interface 2024-03-29T11:11:36.695 INFO:tasks.ceph.osd.1.smithi157.stderr:2024-03-29T11:11:36.717+0000 7fc38d813640 -1 Falling back to public interface 2024-03-29T11:11:36.696 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-03-29T11:11:36.717+0000 7fad3f40f640 -1 Falling back to public interface 2024-03-29T11:11:36.700 INFO:tasks.ceph.osd.0.smithi157.stderr:2024-03-29T11:11:36.722+0000 7fcbdfa1c640 -1 Falling back to public interface 2024-03-29T11:11:36.703 INFO:tasks.ceph.osd.5.smithi195.stderr:2024-03-29T11:11:36.725+0000 7f5207c11640 -1 Falling back to public interface 2024-03-29T11:11:36.711 INFO:tasks.ceph.osd.6.smithi195.stderr:2024-03-29T11:11:36.733+0000 7efd55af9640 -1 Falling back to public interface 2024-03-29T11:11:36.761 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:36.782+0000 7f1690c32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:11:36.830 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:36.852+0000 7fc60e83c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:11:37.101 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:37.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.214+0000 7f40fa940640 10 monclient: get_monmap_and_config 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.214+0000 7f40fa940640 10 monclient: build_initial_monmap 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.214+0000 7f40fa940640 10 monclient: monmap: 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:37.215836+0000 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:37.215836+0000 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:37.193 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:37.201 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.215+0000 7f40fa940640 1 Processor -- start 2024-03-29T11:11:37.201 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.215+0000 7f40fa940640 1 -- start start 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.215+0000 7f40fa940640 10 monclient: init 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 10 monclient(hunting): picked mon.noname-c con 0x7f40f41457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 10 monclient(hunting): picked mon.noname-b con 0x7f40f413f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 10 monclient(hunting): picked mon.noname-a con 0x7f40f40a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f40aaef0 con 0x7f40f40a9d70 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f4145bd0 con 0x7f40f41457f0 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f4145eb0 con 0x7f40f413f650 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 10 monclient(hunting): authenticate will time out at 909.413130s 2024-03-29T11:11:37.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f37fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f40f413f650 0x7f40f40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:59594/0 (socket says 172.21.15.157:59594) 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f37fe640 1 -- 172.21.15.157:0/2660723370 learned_addr learned my addr 172.21.15.157:0/2660723370 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f8edd640 1 --1- 172.21.15.157:0/2660723370 >> v1:172.21.15.157:6789/0 conn(0x7f40f40a9d70 0x7f40f414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:40702/0 (socket says 172.21.15.157:40702) 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914548059 0 0) 0x7f40f4145eb0 con 0x7f40f413f650 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40dc003520 con 0x7f40f413f650 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316833739 0 0) 0x7f40f40aaef0 con 0x7f40f40a9d70 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:11:37.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f4145eb0 con 0x7f40f40a9d70 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958840444 0 0) 0x7f40f4145bd0 con 0x7f40f41457f0 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f40aaef0 con 0x7f40f41457f0 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2398924511 0 0) 0x7f40dc003520 con 0x7f40f413f650 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 >> v1:172.21.15.195:6789/0 conn(0x7f40f41457f0 legacy=0x7f40f413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 >> v1:172.21.15.157:6789/0 conn(0x7f40f40a9d70 legacy=0x7f40f414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:11:37.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4146340 con 0x7f40f413f650 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f40e4001690 con 0x7f40f413f650 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 5 monclient: authenticate success, global_id 4193 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40f2ffd640 10 monclient: dump: 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:37.205 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.216+0000 7f40fa940640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40e40017f0 con 0x7f40f413f650 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40f2ffd640 1 -- 172.21.15.157:0/2660723370 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f40e4001c40 con 0x7f40f413f650 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40f2ffd640 10 monclient: dump: 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.206 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40fa940640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40fa940640 4 set_mon_vals no callback set 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40fa940640 10 monclient: shutdown 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40fa940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40fa940640 1 -- 172.21.15.157:0/2660723370 >> v1:172.21.15.157:6790/0 conn(0x7f40f413f650 legacy=0x7f40f40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40fa940640 1 -- 172.21.15.157:0/2660723370 shutdown_connections 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40fa940640 1 -- 172.21.15.157:0/2660723370 >> 172.21.15.157:0/2660723370 conn(0x7f40f4066a00 msgr2=0x7f40f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.217+0000 7f40fa940640 1 -- 172.21.15.157:0/2660723370 shutdown_connections 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.218+0000 7f40fa940640 1 -- 172.21.15.157:0/2660723370 wait complete. 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.218+0000 7f40fa940640 10 monclient: build_initial_monmap 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.218+0000 7f40fa940640 10 monclient: monmap: 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.218+0000 7f40fa940640 1 librados: starting msgr at 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.218+0000 7f40fa940640 1 librados: starting objecter 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.218+0000 7f40fa940640 1 Processor -- start 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 1 -- start start 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 1 librados: setting wanted keys 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 1 librados: calling monclient init 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 10 monclient: init 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 10 monclient(hunting): picked mon.noname-c con 0x7f40f40a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 10 monclient(hunting): picked mon.noname-b con 0x7f40f413f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 10 monclient(hunting): picked mon.noname-a con 0x7f40f41457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f41db460 con 0x7f40f41457f0 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f41db740 con 0x7f40f413f650 2024-03-29T11:11:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f4142d20 con 0x7f40f40a9d70 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40fa940640 10 monclient(hunting): authenticate will time out at 909.416616s 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40f3fff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f40f40a9d70 0x7f40f41dad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:59596/0 (socket says 172.21.15.157:59596) 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40f3fff640 1 -- 172.21.15.157:0/3767964621 learned_addr learned my addr 172.21.15.157:0/3767964621 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40f0ff9640 10 client.?.objecter ms_handle_connect 0x7f40f40a9d70 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40f0ff9640 10 client.?.objecter ms_handle_connect 0x7f40f41457f0 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.219+0000 7f40f0ff9640 10 client.?.objecter ms_handle_connect 0x7f40f413f650 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633708373 0 0) 0x7f40f4142d20 con 0x7f40f40a9d70 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): my global_id is 4196 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e0004eb0 con 0x7f40f40a9d70 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266965721 0 0) 0x7f40f41db460 con 0x7f40f41457f0 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f4142d20 con 0x7f40f41457f0 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510952865 0 0) 0x7f40f41db740 con 0x7f40f413f650 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:11:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f41db460 con 0x7f40f413f650 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1557541315 0 0) 0x7f40e0004eb0 con 0x7f40f40a9d70 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40f41db740 con 0x7f40f40a9d70 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f40e8001970 con 0x7f40f40a9d70 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:37.212 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2437578910 0 0) 0x7f40f4142d20 con 0x7f40f41457f0 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40e0004eb0 con 0x7f40f41457f0 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f40ec001b60 con 0x7f40f41457f0 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.213 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1684490627 0 0) 0x7f40f41db460 con 0x7f40f413f650 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40f4142d20 con 0x7f40f413f650 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f40e4001b00 con 0x7f40f413f650 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:37.214 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 935128424 0 0) 0x7f40f41db740 con 0x7f40f40a9d70 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 >> v1:172.21.15.195:6789/0 conn(0x7f40f413f650 legacy=0x7f40f40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 >> v1:172.21.15.157:6789/0 conn(0x7f40f41457f0 legacy=0x7f40f41d75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 monclient: found mon.c 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:37.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4142120 con 0x7f40f40a9d70 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40fa940640 5 monclient: authenticate success, global_id 4196 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40fa940640 10 monclient: _renew_subs 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40fa940640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40fa940640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f4142370 con 0x7f40f40a9d70 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40fa940640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40fa940640 10 monclient: _renew_subs 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40fa940640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40fa940640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f40b15e0 con 0x7f40f40a9d70 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40fa940640 1 librados: init done 2024-03-29T11:11:37.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40e8001c50 con 0x7f40f40a9d70 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f40e8003a60 con 0x7f40f40a9d70 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 monclient: dump: 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:37.217 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f27fc640 4 set_mon_vals no callback set 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f40e8004010 con 0x7f40f40a9d70 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f40e8006ab0 con 0x7f40f40a9d70 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 10 client.4196.objecter ms_dispatch 0x7f40f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 3 client.4196.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 3 client.4196.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.220+0000 7f40f0ff9640 20 client.4196.objecter dump_active .. 0 homeless 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.221+0000 7f40fa940640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.221+0000 7f40fa940640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.221+0000 7f40fa940640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.221+0000 7f40fa940640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d4005080 con 0x7f40f40a9d70 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.223+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f40e80063e0 con 0x7f40f40a9d70 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.223+0000 7f40f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:37.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.223+0000 7f40f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:37.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.285+0000 7f40fa940640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:37.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.285+0000 7f40fa940640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:37.264 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.285+0000 7f40fa940640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:37.264 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.285+0000 7f40fa940640 1 -- 172.21.15.157:0/3767964621 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40d4005330 con 0x7f40f40a9d70 2024-03-29T11:11:37.264 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40f0ff9640 1 -- 172.21.15.157:0/3767964621 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f40e8006580 con 0x7f40f40a9d70 2024-03-29T11:11:37.264 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:37.264 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:37.265 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:11:37.265 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40fa940640 10 librados: watch_flush enter 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40fa940640 10 librados: watch_flush exit 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40fa940640 20 client.4196.objecter shutdown clearing up homeless session... 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40fa940640 10 client.4196.objecter successfully canceled tick 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40fa940640 10 monclient: shutdown 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40fa940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40fa940640 1 -- 172.21.15.157:0/3767964621 >> v1:172.21.15.157:6790/0 conn(0x7f40f40a9d70 legacy=0x7f40f41dad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40fa940640 1 -- 172.21.15.157:0/3767964621 shutdown_connections 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.286+0000 7f40fa940640 1 -- 172.21.15.157:0/3767964621 >> 172.21.15.157:0/3767964621 conn(0x7f40f4066a00 msgr2=0x7f40f413aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.287+0000 7f40fa940640 1 -- 172.21.15.157:0/3767964621 shutdown_connections 2024-03-29T11:11:37.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.287+0000 7f40fa940640 1 -- 172.21.15.157:0/3767964621 wait complete. 2024-03-29T11:11:37.267 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:37.287+0000 7f40fa940640 1 librados: shutdown 2024-03-29T11:11:37.274 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:37.581 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:37.603+0000 7f1690c32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:11:37.642 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:37.664+0000 7fc60e83c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:11:37.854 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:37.876+0000 7f1690c32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:11:37.906 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:37.927+0000 7f1690c32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:11:37.917 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:37.939+0000 7fc60e83c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:11:37.968 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:37.990+0000 7fc60e83c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:11:38.019 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:38.041+0000 7f1690c32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:11:38.090 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-03-29T11:11:38.111+0000 7fc60e83c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:11:39.032 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:39.054+0000 7f16311dc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:11:39.033 INFO:tasks.ceph.mgr.y.smithi157.stderr:2024-03-29T11:11:39.055+0000 7f16311dc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:11:40.152 INFO:tasks.ceph.osd.4.smithi195.stderr:2024-03-29T11:11:40.174+0000 7f5043c11640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:11:40.177 INFO:tasks.ceph.osd.2.smithi157.stderr:2024-03-29T11:11:40.199+0000 7f7505a66640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:11:40.189 INFO:tasks.ceph.osd.0.smithi157.stderr:2024-03-29T11:11:40.210+0000 7fcbdfa1c640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:11:40.210 INFO:tasks.ceph.osd.5.smithi195.stderr:2024-03-29T11:11:40.232+0000 7f5207c11640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:11:40.216 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-03-29T11:11:40.238+0000 7fad3f40f640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:11:40.223 INFO:tasks.ceph.osd.6.smithi195.stderr:2024-03-29T11:11:40.245+0000 7efd55af9640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:11:40.229 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-03-29T11:11:40.251+0000 7fcbe7613640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:11:40.231 INFO:tasks.ceph.osd.1.smithi157.stderr:2024-03-29T11:11:40.253+0000 7fc38d813640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:11:44.083 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:44.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.254+0000 7f89eda86640 10 monclient: get_monmap_and_config 2024-03-29T11:11:44.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.254+0000 7f89eda86640 10 monclient: build_initial_monmap 2024-03-29T11:11:44.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.254+0000 7f89eda86640 10 monclient: monmap: 2024-03-29T11:11:44.232 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:11:44.232 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:44.233 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:44.255191+0000 2024-03-29T11:11:44.233 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:44.255191+0000 2024-03-29T11:11:44.233 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:44.233 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:44.233 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:44.233 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:44.233 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:44.233 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:44.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.255+0000 7f89eda86640 1 Processor -- start 2024-03-29T11:11:44.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.255+0000 7f89eda86640 1 -- start start 2024-03-29T11:11:44.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.255+0000 7f89eda86640 10 monclient: init 2024-03-29T11:11:44.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89eda86640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:44.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89eda86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:44.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89eda86640 10 monclient(hunting): picked mon.noname-c con 0x7f89e80b0770 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:44.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89eda86640 10 monclient(hunting): picked mon.noname-b con 0x7f89e814a700 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:44.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89eda86640 10 monclient(hunting): picked mon.noname-a con 0x7f89e80ac5a0 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:44.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89eda86640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80aba70 con 0x7f89e80ac5a0 2024-03-29T11:11:44.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89eda86640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80aecb0 con 0x7f89e80b0770 2024-03-29T11:11:44.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89eda86640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80af150 con 0x7f89e814a700 2024-03-29T11:11:44.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89eda86640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:44.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89eda86640 10 monclient(hunting): authenticate will time out at 916.454058s 2024-03-29T11:11:44.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89e6ffd640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f89e814a700 0x7f89e80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:38468/0 (socket says 172.21.15.157:38468) 2024-03-29T11:11:44.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.257+0000 7f89e6ffd640 1 -- 172.21.15.157:0/1994239706 learned_addr learned my addr 172.21.15.157:0/1994239706 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:44.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085763463 0 0) 0x7f89e80aecb0 con 0x7f89e80b0770 2024-03-29T11:11:44.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:44.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:44.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:44.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:11:44.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d0003520 con 0x7f89e80b0770 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208153170 0 0) 0x7f89e80af150 con 0x7f89e814a700 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.258+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e80aecb0 con 0x7f89e814a700 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760511688 0 0) 0x7f89e80aba70 con 0x7f89e80ac5a0 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient(hunting): my global_id is 4236 2024-03-29T11:11:44.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d00039d0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191253320 0 0) 0x7f89d0003520 con 0x7f89e80b0770 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 >> v1:172.21.15.157:6790/0 conn(0x7f89e814a700 legacy=0x7f89e80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 >> v1:172.21.15.157:6789/0 conn(0x7f89e80ac5a0 legacy=0x7f89e8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 1 monclient: found mon.noname-c 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e80ac9d0 con 0x7f89e80b0770 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: _finish_auth 0 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f89d80016b0 con 0x7f89e80b0770 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:44.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: dump: 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89eda86640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89eda86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89d8001810 con 0x7f89e80b0770 2024-03-29T11:11:44.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 1 -- 172.21.15.157:0/1994239706 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f89d8001c20 con 0x7f89e80b0770 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89e67fc640 10 monclient: dump: 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89eda86640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:44.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.259+0000 7f89eda86640 4 set_mon_vals no callback set 2024-03-29T11:11:44.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.260+0000 7f89eda86640 10 monclient: shutdown 2024-03-29T11:11:44.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.260+0000 7f89eda86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:44.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.260+0000 7f89eda86640 1 -- 172.21.15.157:0/1994239706 >> v1:172.21.15.195:6789/0 conn(0x7f89e80b0770 legacy=0x7f89e8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:44.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.261+0000 7f89eda86640 1 -- 172.21.15.157:0/1994239706 shutdown_connections 2024-03-29T11:11:44.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.261+0000 7f89eda86640 1 -- 172.21.15.157:0/1994239706 >> 172.21.15.157:0/1994239706 conn(0x7f89e806b660 msgr2=0x7f89e806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:44.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.261+0000 7f89eda86640 1 -- 172.21.15.157:0/1994239706 shutdown_connections 2024-03-29T11:11:44.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.261+0000 7f89eda86640 1 -- 172.21.15.157:0/1994239706 wait complete. 2024-03-29T11:11:44.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.262+0000 7f89eda86640 10 monclient: build_initial_monmap 2024-03-29T11:11:44.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.262+0000 7f89eda86640 10 monclient: monmap: 2024-03-29T11:11:44.250 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:11:44.250 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.262+0000 7f89eda86640 1 librados: starting msgr at 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.262+0000 7f89eda86640 1 librados: starting objecter 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.263+0000 7f89eda86640 1 Processor -- start 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.263+0000 7f89eda86640 1 -- start start 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.263+0000 7f89eda86640 1 librados: setting wanted keys 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.263+0000 7f89eda86640 1 librados: calling monclient init 2024-03-29T11:11:44.251 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.263+0000 7f89eda86640 10 monclient: init 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89eda86640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89eda86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89eda86640 10 monclient(hunting): picked mon.noname-b con 0x7f89e80ac5a0 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89eda86640 10 monclient(hunting): picked mon.noname-a con 0x7f89e80b0770 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89eda86640 10 monclient(hunting): picked mon.noname-c con 0x7f89e814a700 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89eda86640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e81ec790 con 0x7f89e80b0770 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89eda86640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e807f550 con 0x7f89e80ac5a0 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89eda86640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e807e560 con 0x7f89e814a700 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89eda86640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89eda86640 10 monclient(hunting): authenticate will time out at 916.461424s 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89e77fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f89e80ac5a0 0x7f89e807de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:55060/0 (socket says 172.21.15.157:55060) 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.264+0000 7f89e77fe640 1 -- 172.21.15.157:0/16715647 learned_addr learned my addr 172.21.15.157:0/16715647 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 10 client.?.objecter ms_handle_connect 0x7f89e80ac5a0 2024-03-29T11:11:44.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398220739 0 0) 0x7f89e807f550 con 0x7f89e80ac5a0 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d4004eb0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 10 client.?.objecter ms_handle_connect 0x7f89e80b0770 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 10 client.?.objecter ms_handle_connect 0x7f89e814a700 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.265+0000 7f89cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2784867503 0 0) 0x7f89d4004eb0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89e807f550 con 0x7f89e80ac5a0 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f89d8001be0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient(hunting): dump: 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:44.254 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440761233 0 0) 0x7f89e81ec790 con 0x7f89e80b0770 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d4004eb0 con 0x7f89e80b0770 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2055469390 0 0) 0x7f89e807f550 con 0x7f89e80ac5a0 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 >> v1:172.21.15.157:6790/0 conn(0x7f89e814a700 legacy=0x7f89e81e9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 >> v1:172.21.15.157:6789/0 conn(0x7f89e80b0770 legacy=0x7f89e81ecc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient: _finish_hunting 0 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 1 monclient: found mon.b 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e807aff0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient: _finish_auth 0 2024-03-29T11:11:44.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 10 monclient: _check_auth_tickets 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89eda86640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89eda86640 10 monclient: _renew_subs 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89eda86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89eda86640 1 -- 172.21.15.157:0/16715647 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e81ed3a0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89eda86640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89eda86640 10 monclient: _renew_subs 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89eda86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89eda86640 1 -- 172.21.15.157:0/16715647 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e807d3b0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.266+0000 7f89eda86640 1 librados: init done 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.267+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89d8001210 con 0x7f89e80ac5a0 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.267+0000 7f89cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.267+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f89d8003a80 con 0x7f89e80ac5a0 2024-03-29T11:11:44.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.267+0000 7f89cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.267+0000 7f89cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.267+0000 7f89cffff640 10 monclient: dump: 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.267+0000 7f89e5ffb640 4 set_mon_vals no callback set 2024-03-29T11:11:44.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.268+0000 7f89eda86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.268+0000 7f89eda86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.268+0000 7f89eda86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.269+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f89d8003be0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.270+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f89d8084ab0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.270+0000 7f89cffff640 10 client.4240.objecter ms_dispatch 0x7f89e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.270+0000 7f89cffff640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.270+0000 7f89cffff640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.270+0000 7f89cffff640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.268+0000 7f89eda86640 1 -- 172.21.15.157:0/16715647 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c8005080 con 0x7f89e80ac5a0 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.270+0000 7f89e6ffd640 10 monclient: get_auth_request con 0x7f89d406ddb0 auth_method 0 2024-03-29T11:11:44.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.271+0000 7f89cffff640 10 client.4240.objecter ms_handle_connect 0x7f89d406ddb0 2024-03-29T11:11:44.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.277+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89d80501c0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.277+0000 7f89cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:44.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.277+0000 7f89cffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:44.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.678+0000 7f89eda86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:44.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.678+0000 7f89eda86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:44.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.678+0000 7f89eda86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:11:44.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.678+0000 7f89eda86640 1 -- 172.21.15.157:0/16715647 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89c8005330 con 0x7f89e80ac5a0 2024-03-29T11:11:44.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.679+0000 7f89cffff640 1 -- 172.21.15.157:0/16715647 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f89d80522c0 con 0x7f89e80ac5a0 2024-03-29T11:11:44.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.679+0000 7f89cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:44.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.679+0000 7f89cffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:44.658 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:11:44.658 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:44.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.683+0000 7f89eda86640 10 librados: watch_flush enter 2024-03-29T11:11:44.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.683+0000 7f89eda86640 10 librados: watch_flush exit 2024-03-29T11:11:44.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.683+0000 7f89eda86640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T11:11:44.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.683+0000 7f89eda86640 10 client.4240.objecter successfully canceled tick 2024-03-29T11:11:44.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.683+0000 7f89eda86640 1 -- 172.21.15.157:0/16715647 >> v1:172.21.15.157:6816/1747058247 conn(0x7f89d406ddb0 legacy=0x7f89d4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:44.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.683+0000 7f89eda86640 10 monclient: shutdown 2024-03-29T11:11:44.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.683+0000 7f89eda86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:44.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.683+0000 7f89eda86640 1 -- 172.21.15.157:0/16715647 >> v1:172.21.15.195:6789/0 conn(0x7f89e80ac5a0 legacy=0x7f89e807de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:44.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.684+0000 7f89eda86640 1 -- 172.21.15.157:0/16715647 shutdown_connections 2024-03-29T11:11:44.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.684+0000 7f89eda86640 1 -- 172.21.15.157:0/16715647 >> 172.21.15.157:0/16715647 conn(0x7f89e806b660 msgr2=0x7f89e8149530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:44.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.685+0000 7f89eda86640 1 -- 172.21.15.157:0/16715647 shutdown_connections 2024-03-29T11:11:44.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.685+0000 7f89eda86640 1 -- 172.21.15.157:0/16715647 wait complete. 2024-03-29T11:11:44.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:44.685+0000 7f89eda86640 1 librados: shutdown 2024-03-29T11:11:44.680 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:51.487 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:51.647 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.668+0000 7f5ed71f4640 10 monclient: get_monmap_and_config 2024-03-29T11:11:51.647 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.668+0000 7f5ed71f4640 10 monclient: build_initial_monmap 2024-03-29T11:11:51.647 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.668+0000 7f5ed71f4640 10 monclient: monmap: 2024-03-29T11:11:51.647 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:11:51.647 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:51.647 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:51.668923+0000 2024-03-29T11:11:51.647 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:51.668923+0000 2024-03-29T11:11:51.647 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:51.647 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:51.648 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:51.648 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:51.648 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:51.648 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:51.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.670+0000 7f5ed71f4640 1 Processor -- start 2024-03-29T11:11:51.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.670+0000 7f5ed71f4640 1 -- start start 2024-03-29T11:11:51.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.670+0000 7f5ed71f4640 10 monclient: init 2024-03-29T11:11:51.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.670+0000 7f5ed71f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:51.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.670+0000 7f5ed71f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:51.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ed71f4640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed01457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:51.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ed4f90640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f5ed01457f0 0x7f5ed013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:40282/0 (socket says 172.21.15.157:40282) 2024-03-29T11:11:51.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ed4f90640 1 -- 172.21.15.157:0/3940299477 learned_addr learned my addr 172.21.15.157:0/3940299477 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:51.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ed71f4640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed013f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:51.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ed71f4640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed00a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ed71f4640 1 -- 172.21.15.157:0/3940299477 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed00aaef0 con 0x7f5ed00a9d70 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ed71f4640 1 -- 172.21.15.157:0/3940299477 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed0145bd0 con 0x7f5ed013f650 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ed71f4640 1 -- 172.21.15.157:0/3940299477 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed0145eb0 con 0x7f5ed01457f0 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ed71f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ed71f4640 10 monclient(hunting): authenticate will time out at 923.867179s 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521460034 0 0) 0x7f5ed0145eb0 con 0x7f5ed01457f0 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc003520 con 0x7f5ed01457f0 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777423053 0 0) 0x7f5ed00aaef0 con 0x7f5ed00a9d70 2024-03-29T11:11:51.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:51.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:51.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:51.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:11:51.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.671+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed0145eb0 con 0x7f5ed00a9d70 2024-03-29T11:11:51.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2629949710 0 0) 0x7f5ebc003520 con 0x7f5ed01457f0 2024-03-29T11:11:51.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 >> v1:172.21.15.195:6789/0 conn(0x7f5ed013f650 legacy=0x7f5ed00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:51.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 >> v1:172.21.15.157:6789/0 conn(0x7f5ed00a9d70 legacy=0x7f5ed014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 1 monclient: found mon.noname-b 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0146340 con 0x7f5ed01457f0 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5eb8001690 con 0x7f5ed01457f0 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:51.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ecf7fe640 10 monclient: dump: 2024-03-29T11:11:51.653 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:51.653 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:51.653 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:51.654 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:51.654 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:51.654 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:51.654 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:51.654 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:51.654 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:51.654 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:51.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ed71f4640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:11:51.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.672+0000 7f5ed71f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:51.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.673+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eb80017f0 con 0x7f5ed01457f0 2024-03-29T11:11:51.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.673+0000 7f5ecf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:51.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.673+0000 7f5ecf7fe640 1 -- 172.21.15.157:0/3940299477 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5eb8001c40 con 0x7f5ed01457f0 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.673+0000 7f5ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.673+0000 7f5ecf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.673+0000 7f5ecf7fe640 10 monclient: dump: 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:51.662 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:51.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.673+0000 7f5ed71f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:51.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.673+0000 7f5ed71f4640 4 set_mon_vals no callback set 2024-03-29T11:11:51.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.674+0000 7f5ed71f4640 10 monclient: shutdown 2024-03-29T11:11:51.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.674+0000 7f5ed71f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:51.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.674+0000 7f5ed71f4640 1 -- 172.21.15.157:0/3940299477 >> v1:172.21.15.157:6790/0 conn(0x7f5ed01457f0 legacy=0x7f5ed013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.674+0000 7f5ed71f4640 1 -- 172.21.15.157:0/3940299477 shutdown_connections 2024-03-29T11:11:51.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.674+0000 7f5ed71f4640 1 -- 172.21.15.157:0/3940299477 >> 172.21.15.157:0/3940299477 conn(0x7f5ed0066a00 msgr2=0x7f5ed0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:51.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.674+0000 7f5ed71f4640 1 -- 172.21.15.157:0/3940299477 shutdown_connections 2024-03-29T11:11:51.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.674+0000 7f5ed71f4640 1 -- 172.21.15.157:0/3940299477 wait complete. 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.675+0000 7f5ed71f4640 10 monclient: build_initial_monmap 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.675+0000 7f5ed71f4640 10 monclient: monmap: 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.675+0000 7f5ed71f4640 1 librados: starting msgr at 2024-03-29T11:11:51.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.676+0000 7f5ed71f4640 1 librados: starting objecter 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.676+0000 7f5ed71f4640 1 Processor -- start 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.676+0000 7f5ed71f4640 1 -- start start 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.676+0000 7f5ed71f4640 1 librados: setting wanted keys 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.676+0000 7f5ed71f4640 1 librados: calling monclient init 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.676+0000 7f5ed71f4640 10 monclient: init 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed71f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed71f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed71f4640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed00a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed71f4640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed013f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed71f4640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed01457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed71f4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed0144620 con 0x7f5ed00a9d70 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed71f4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed0141370 con 0x7f5ed013f650 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed71f4640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed0141650 con 0x7f5ed01457f0 2024-03-29T11:11:51.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed71f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed71f4640 10 monclient(hunting): authenticate will time out at 923.873657s 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.677+0000 7f5ed4f90640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f5ed00a9d70 0x7f5ed013fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:38780/0 (socket says 172.21.15.157:38780) 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ed4f90640 1 -- 172.21.15.157:0/1873401897 learned_addr learned my addr 172.21.15.157:0/1873401897 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 client.?.objecter ms_handle_connect 0x7f5ed01457f0 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 client.?.objecter ms_handle_connect 0x7f5ed013f650 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054060631 0 0) 0x7f5ed0141650 con 0x7f5ed01457f0 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb0004eb0 con 0x7f5ed01457f0 2024-03-29T11:11:51.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 client.?.objecter ms_handle_connect 0x7f5ed00a9d70 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982179882 0 0) 0x7f5ed0141370 con 0x7f5ed013f650 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed0141650 con 0x7f5ed013f650 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517589013 0 0) 0x7f5ed0144620 con 0x7f5ed00a9d70 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.678+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed0141370 con 0x7f5ed00a9d70 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.679+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3080525668 0 0) 0x7f5eb0004eb0 con 0x7f5ed01457f0 2024-03-29T11:11:51.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.679+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ed0144620 con 0x7f5ed01457f0 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.679+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5ec8001a30 con 0x7f5ed01457f0 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.679+0000 7f5ecd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.679+0000 7f5ecd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.679+0000 7f5ecd7fa640 10 monclient(hunting): dump: 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:51.668 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.679+0000 7f5ecd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.679+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 291040700 0 0) 0x7f5ed0144620 con 0x7f5ed01457f0 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.679+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 >> v1:172.21.15.195:6789/0 conn(0x7f5ed013f650 legacy=0x7f5ed01dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.679+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 >> v1:172.21.15.157:6789/0 conn(0x7f5ed00a9d70 legacy=0x7f5ed013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ecd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ecd7fa640 1 monclient: found mon.c 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ecd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ecd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed013c650 con 0x7f5ed01457f0 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ecd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ecd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ecd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ed71f4640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:11:51.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ed71f4640 10 monclient: _renew_subs 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ed71f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ed71f4640 1 -- 172.21.15.157:0/1873401897 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed01404b0 con 0x7f5ed01457f0 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ed71f4640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ed71f4640 10 monclient: _renew_subs 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ed71f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ed71f4640 1 -- 172.21.15.157:0/1873401897 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed01452c0 con 0x7f5ed01457f0 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.680+0000 7f5ed71f4640 1 librados: init done 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.682+0000 7f5ed71f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.682+0000 7f5ed71f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.682+0000 7f5ed71f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.682+0000 7f5ed71f4640 1 -- 172.21.15.157:0/1873401897 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed01447d0 con 0x7f5ed01457f0 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.683+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ec8003970 con 0x7f5ed01457f0 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.683+0000 7f5ecd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:51.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.683+0000 7f5eceffd640 4 set_mon_vals no callback set 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.683+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5ec8003c80 con 0x7f5ed01457f0 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.683+0000 7f5ecd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.683+0000 7f5ecd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.683+0000 7f5ecd7fa640 10 monclient: dump: 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:51.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.683+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f5ec8019c50 con 0x7f5ed01457f0 2024-03-29T11:11:51.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.684+0000 7f5ed4f90640 10 monclient: get_auth_request con 0x7f5eb006dc80 auth_method 0 2024-03-29T11:11:51.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.685+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f5ec8083c30 con 0x7f5ed01457f0 2024-03-29T11:11:51.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.685+0000 7f5ecd7fa640 10 client.4250.objecter ms_dispatch 0x7f5ed0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:51.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.685+0000 7f5ecd7fa640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:51.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.685+0000 7f5ecd7fa640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:51.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.685+0000 7f5ecd7fa640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T11:11:51.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.685+0000 7f5ecd7fa640 10 client.4250.objecter ms_handle_connect 0x7f5eb006dc80 2024-03-29T11:11:51.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.691+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ed01447d0 con 0x7f5ed01457f0 2024-03-29T11:11:51.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.691+0000 7f5ecd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:51.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:51.691+0000 7f5ecd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:52.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.031+0000 7f5ed71f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:52.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.031+0000 7f5ed71f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:52.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.031+0000 7f5ed71f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:11:52.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.031+0000 7f5ed71f4640 1 -- 172.21.15.157:0/1873401897 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ed01447d0 con 0x7f5ed01457f0 2024-03-29T11:11:52.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.035+0000 7f5ecd7fa640 1 -- 172.21.15.157:0/1873401897 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f5ed01447d0 con 0x7f5ed01457f0 2024-03-29T11:11:52.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.035+0000 7f5ecd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:52.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.035+0000 7f5ecd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:52.013 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:11:52.013 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:52.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 10 librados: watch_flush enter 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 10 librados: watch_flush exit 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 10 client.4250.objecter successfully canceled tick 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 1 -- 172.21.15.157:0/1873401897 >> v1:172.21.15.157:6816/1747058247 conn(0x7f5eb006dc80 legacy=0x7f5eb0070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 10 monclient: shutdown 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 1 -- 172.21.15.157:0/1873401897 >> v1:172.21.15.157:6790/0 conn(0x7f5ed01457f0 legacy=0x7f5ed01e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 1 -- 172.21.15.157:0/1873401897 shutdown_connections 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 1 -- 172.21.15.157:0/1873401897 >> 172.21.15.157:0/1873401897 conn(0x7f5ed0066a00 msgr2=0x7f5ed00b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 1 -- 172.21.15.157:0/1873401897 shutdown_connections 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 1 -- 172.21.15.157:0/1873401897 wait complete. 2024-03-29T11:11:52.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:52.038+0000 7f5ed71f4640 1 librados: shutdown 2024-03-29T11:11:52.030 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:58.837 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:58.982 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.003+0000 7f4c10167640 10 monclient: get_monmap_and_config 2024-03-29T11:11:58.982 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.003+0000 7f4c10167640 10 monclient: build_initial_monmap 2024-03-29T11:11:58.982 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.003+0000 7f4c10167640 10 monclient: monmap: 2024-03-29T11:11:58.982 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:11:58.982 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:58.982 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:59.005025+0000 2024-03-29T11:11:58.982 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:59.005025+0000 2024-03-29T11:11:58.982 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:58.983 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:58.983 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:58.983 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:11:58.983 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:11:58.983 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:58.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.005+0000 7f4c10167640 1 Processor -- start 2024-03-29T11:11:58.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.005+0000 7f4c10167640 1 -- start start 2024-03-29T11:11:58.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.005+0000 7f4c10167640 10 monclient: init 2024-03-29T11:11:58.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.006+0000 7f4c10167640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:58.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.006+0000 7f4c10167640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:58.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.006+0000 7f4c10167640 10 monclient(hunting): picked mon.noname-b con 0x7f4c080b1f60 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:58.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.006+0000 7f4c10167640 10 monclient(hunting): picked mon.noname-c con 0x7f4c08071eb0 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:58.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.006+0000 7f4c10167640 10 monclient(hunting): picked mon.noname-a con 0x7f4c08143d10 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:58.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.006+0000 7f4c10167640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0806ef80 con 0x7f4c08143d10 2024-03-29T11:11:58.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.006+0000 7f4c10167640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0806f260 con 0x7f4c08071eb0 2024-03-29T11:11:58.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.006+0000 7f4c10167640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c080b40c0 con 0x7f4c080b1f60 2024-03-29T11:11:58.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.006+0000 7f4c10167640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:58.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.006+0000 7f4c10167640 10 monclient(hunting): authenticate will time out at 931.203211s 2024-03-29T11:11:58.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.007+0000 7f4c0d702640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4c08071eb0 0x7f4c08146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:46934/0 (socket says 172.21.15.157:46934) 2024-03-29T11:11:58.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.007+0000 7f4c0d702640 1 -- 172.21.15.157:0/2011706620 learned_addr learned my addr 172.21.15.157:0/2011706620 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698381162 0 0) 0x7f4c0806ef80 con 0x7f4c08143d10 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): _negotiate 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf4003520 con 0x7f4c08143d10 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762949649 0 0) 0x7f4c0806f260 con 0x7f4c08071eb0 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): _negotiate 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:11:58.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0806ef80 con 0x7f4c08071eb0 2024-03-29T11:11:58.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276287733 0 0) 0x7f4c080b40c0 con 0x7f4c080b1f60 2024-03-29T11:11:58.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): _negotiate 2024-03-29T11:11:58.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:58.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:58.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:11:58.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0806f260 con 0x7f4c080b1f60 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2376189118 0 0) 0x7f4bf4003520 con 0x7f4c08143d10 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 >> v1:172.21.15.157:6790/0 conn(0x7f4c080b1f60 legacy=0x7f4c08143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 >> v1:172.21.15.195:6789/0 conn(0x7f4c08071eb0 legacy=0x7f4c08146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: _finish_hunting 0 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 monclient: found mon.noname-a 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c08155d70 con 0x7f4c08143d10 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: _finish_auth 0 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: _check_auth_tickets 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4bf8001690 con 0x7f4c08143d10 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:58.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: dump: 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bf80017f0 con 0x7f4c08143d10 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:58.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 1 -- 172.21.15.157:0/2011706620 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4bf8001c00 con 0x7f4c08143d10 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.011+0000 7f4c0cf01640 10 monclient: dump: 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.012+0000 7f4c10167640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:11:58.993 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.012+0000 7f4c10167640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:58.994 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.012+0000 7f4c10167640 4 set_mon_vals no callback set 2024-03-29T11:11:58.997 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.013+0000 7f4c10167640 10 monclient: shutdown 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.013+0000 7f4c10167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.013+0000 7f4c10167640 1 -- 172.21.15.157:0/2011706620 >> v1:172.21.15.157:6789/0 conn(0x7f4c08143d10 legacy=0x7f4c08146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.013+0000 7f4c10167640 1 -- 172.21.15.157:0/2011706620 shutdown_connections 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.013+0000 7f4c10167640 1 -- 172.21.15.157:0/2011706620 >> 172.21.15.157:0/2011706620 conn(0x7f4c08066a00 msgr2=0x7f4c08066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.014+0000 7f4c10167640 1 -- 172.21.15.157:0/2011706620 shutdown_connections 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.014+0000 7f4c10167640 1 -- 172.21.15.157:0/2011706620 wait complete. 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.015+0000 7f4c10167640 10 monclient: build_initial_monmap 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.015+0000 7f4c10167640 10 monclient: monmap: 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:11:58.998 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.016+0000 7f4c10167640 1 librados: starting msgr at 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.016+0000 7f4c10167640 1 librados: starting objecter 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.016+0000 7f4c10167640 1 Processor -- start 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.016+0000 7f4c10167640 1 -- start start 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.016+0000 7f4c10167640 1 librados: setting wanted keys 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.016+0000 7f4c10167640 1 librados: calling monclient init 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.016+0000 7f4c10167640 10 monclient: init 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.017+0000 7f4c10167640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:58.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.017+0000 7f4c10167640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:11:59.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.017+0000 7f4c10167640 10 monclient(hunting): picked mon.noname-b con 0x7f4c080b1f60 addr v1:172.21.15.195:6789/0 2024-03-29T11:11:59.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.019+0000 7f4c0df03640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4c080b1f60 0x7f4c0813f590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:47036/0 (socket says 172.21.15.157:47036) 2024-03-29T11:11:59.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.019+0000 7f4c0df03640 1 -- 172.21.15.157:0/3125281775 learned_addr learned my addr 172.21.15.157:0/3125281775 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:11:59.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4c10167640 10 monclient(hunting): picked mon.noname-a con 0x7f4c08143d10 addr v1:172.21.15.157:6789/0 2024-03-29T11:11:59.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4bfe7fc640 10 client.?.objecter ms_handle_connect 0x7f4c080b1f60 2024-03-29T11:11:59.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4bfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:59.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4bfe7fc640 10 client.?.objecter ms_handle_connect 0x7f4c08143d10 2024-03-29T11:11:59.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4bfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:59.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4c10167640 10 monclient(hunting): picked mon.noname-c con 0x7f4c08149380 addr v1:172.21.15.157:6790/0 2024-03-29T11:11:59.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0814afd0 con 0x7f4c08143d10 2024-03-29T11:11:59.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0813b960 con 0x7f4c080b1f60 2024-03-29T11:11:59.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0814b220 con 0x7f4c08149380 2024-03-29T11:11:59.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4c10167640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:59.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4c10167640 10 monclient(hunting): authenticate will time out at 931.217284s 2024-03-29T11:11:59.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873199447 0 0) 0x7f4c0814afd0 con 0x7f4c08143d10 2024-03-29T11:11:59.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4bfe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:59.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4bfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:59.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4bfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:59.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4bfe7fc640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:11:59.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.020+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bec004eb0 con 0x7f4c08143d10 2024-03-29T11:11:59.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.021+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2479979525 0 0) 0x7f4bec004eb0 con 0x7f4c08143d10 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.021+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c0814afd0 con 0x7f4c08143d10 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.021+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4c00001be0 con 0x7f4c08143d10 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.021+0000 7f4bfe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.021+0000 7f4bfe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.021+0000 7f4bfe7fc640 10 monclient(hunting): dump: 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:59.003 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:59.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.021+0000 7f4bfe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:59.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.021+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2385200748 0 0) 0x7f4c0814afd0 con 0x7f4c08143d10 2024-03-29T11:11:59.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.021+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 >> v1:172.21.15.157:6790/0 conn(0x7f4c08149380 legacy=0x7f4c0813b250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:59.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.023+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 >> v1:172.21.15.195:6789/0 conn(0x7f4c080b1f60 legacy=0x7f4c0813f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:59.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.023+0000 7f4bfe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:59.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.023+0000 7f4bfe7fc640 1 monclient: found mon.a 2024-03-29T11:11:59.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.023+0000 7f4bfe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:59.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4bfe7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:11:59.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0814b500 con 0x7f4c08143d10 2024-03-29T11:11:59.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4bfe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:11:59.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4bfe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:59.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4bfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:59.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4c10167640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:11:59.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4c10167640 10 monclient: _renew_subs 2024-03-29T11:11:59.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4c10167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:11:59.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c081497b0 con 0x7f4c08143d10 2024-03-29T11:11:59.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4c10167640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:59.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4c10167640 10 monclient: _renew_subs 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4c10167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c08145630 con 0x7f4c08143d10 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.024+0000 7f4c10167640 1 librados: init done 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4c10167640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4c10167640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4c10167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c00004b60 con 0x7f4c08143d10 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c08075f10 con 0x7f4c08143d10 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4bfe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4c00004ef0 con 0x7f4c08143d10 2024-03-29T11:11:59.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4bfe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4bfe7fc640 10 monclient: dump: 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:11:59.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.025+0000 7f4bfffff640 4 set_mon_vals no callback set 2024-03-29T11:11:59.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.028+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f4c00005050 con 0x7f4c08143d10 2024-03-29T11:11:59.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.029+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f4c00084d20 con 0x7f4c08143d10 2024-03-29T11:11:59.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.029+0000 7f4bfe7fc640 10 client.4251.objecter ms_dispatch 0x7f4c08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:59.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.029+0000 7f4bfe7fc640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:59.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.029+0000 7f4bfe7fc640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:59.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.029+0000 7f4bfe7fc640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:11:59.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.029+0000 7f4c0df03640 10 monclient: get_auth_request con 0x7f4bec06dce0 auth_method 0 2024-03-29T11:11:59.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.030+0000 7f4bfe7fc640 10 client.4251.objecter ms_handle_connect 0x7f4bec06dce0 2024-03-29T11:11:59.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.034+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c00050600 con 0x7f4c08143d10 2024-03-29T11:11:59.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.034+0000 7f4bfe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:59.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.034+0000 7f4bfe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:59.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.359+0000 7f4c10167640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:59.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.359+0000 7f4c10167640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:59.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.359+0000 7f4c10167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:11:59.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.359+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c08074f50 con 0x7f4c08143d10 2024-03-29T11:11:59.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.360+0000 7f4bfe7fc640 1 -- 172.21.15.157:0/3125281775 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f4c00052530 con 0x7f4c08143d10 2024-03-29T11:11:59.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.360+0000 7f4bfe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:59.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.360+0000 7f4bfe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:59.342 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:11:59.342 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.364+0000 7f4c10167640 10 librados: watch_flush enter 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.364+0000 7f4c10167640 10 librados: watch_flush exit 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.364+0000 7f4c10167640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.364+0000 7f4c10167640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.364+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 >> v1:172.21.15.157:6816/1747058247 conn(0x7f4bec06dce0 legacy=0x7f4bec0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.364+0000 7f4c10167640 10 monclient: shutdown 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.364+0000 7f4c10167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.364+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 >> v1:172.21.15.157:6789/0 conn(0x7f4c08143d10 legacy=0x7f4c0813ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.364+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 shutdown_connections 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.364+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 >> 172.21.15.157:0/3125281775 conn(0x7f4c08066a00 msgr2=0x7f4c080b7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:59.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.365+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 shutdown_connections 2024-03-29T11:11:59.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.365+0000 7f4c10167640 1 -- 172.21.15.157:0/3125281775 wait complete. 2024-03-29T11:11:59.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:11:59.365+0000 7f4c10167640 1 librados: shutdown 2024-03-29T11:11:59.359 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:06.168 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:06.320 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.339+0000 7f91b292b640 10 monclient: get_monmap_and_config 2024-03-29T11:12:06.320 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.339+0000 7f91b292b640 10 monclient: build_initial_monmap 2024-03-29T11:12:06.320 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.339+0000 7f91b292b640 10 monclient: monmap: 2024-03-29T11:12:06.320 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:06.320 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:06.320 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:12:06.340308+0000 2024-03-29T11:12:06.320 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:12:06.340308+0000 2024-03-29T11:12:06.321 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:06.321 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:06.321 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:06.321 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:12:06.321 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:12:06.321 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:06.321 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.340+0000 7f91b292b640 1 Processor -- start 2024-03-29T11:12:06.321 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.340+0000 7f91b292b640 1 -- start start 2024-03-29T11:12:06.321 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.340+0000 7f91b292b640 10 monclient: init 2024-03-29T11:12:06.322 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.341+0000 7f91b292b640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:06.323 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.341+0000 7f91b292b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:06.323 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.341+0000 7f91b292b640 10 monclient(hunting): picked mon.noname-b con 0x7f91ac071de0 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:06.323 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.341+0000 7f91b292b640 10 monclient(hunting): picked mon.noname-a con 0x7f91ac0721c0 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:06.323 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.341+0000 7f91b292b640 10 monclient(hunting): picked mon.noname-c con 0x7f91ac145ae0 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:06.323 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.341+0000 7f91b292b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac145650 con 0x7f91ac0721c0 2024-03-29T11:12:06.323 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.341+0000 7f91b292b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac0ae260 con 0x7f91ac145ae0 2024-03-29T11:12:06.323 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.341+0000 7f91b292b640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac1420d0 con 0x7f91ac071de0 2024-03-29T11:12:06.323 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.341+0000 7f91b292b640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:06.323 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.341+0000 7f91b292b640 10 monclient(hunting): authenticate will time out at 938.538544s 2024-03-29T11:12:06.325 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91b0ec8640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f91ac145ae0 0x7f91ac145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:37936/0 (socket says 172.21.15.157:37936) 2024-03-29T11:12:06.325 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91b0ec8640 1 -- 172.21.15.157:0/2621612562 learned_addr learned my addr 172.21.15.157:0/2621612562 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:06.325 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857105153 0 0) 0x7f91ac0ae260 con 0x7f91ac145ae0 2024-03-29T11:12:06.325 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:06.325 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:06.325 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:06.325 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:12:06.325 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9198003520 con 0x7f91ac145ae0 2024-03-29T11:12:06.326 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809214642 0 0) 0x7f91ac145650 con 0x7f91ac0721c0 2024-03-29T11:12:06.326 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:06.326 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:06.326 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:06.326 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:12:06.326 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.345+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ac0ae260 con 0x7f91ac0721c0 2024-03-29T11:12:06.326 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.346+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4005849514 0 0) 0x7f91ac0ae260 con 0x7f91ac0721c0 2024-03-29T11:12:06.326 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.346+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 >> v1:172.21.15.157:6790/0 conn(0x7f91ac071de0 legacy=0x7f91ac0aed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:06.327 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.348+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 >> v1:172.21.15.195:6789/0 conn(0x7f91ac145ae0 legacy=0x7f91ac145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.327 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.348+0000 7f91aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:06.327 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.348+0000 7f91aaffd640 1 monclient: found mon.noname-a 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.348+0000 7f91aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.348+0000 7f91aaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac0b0590 con 0x7f91ac0721c0 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91b292b640 5 monclient: authenticate success, global_id 4254 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91b292b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f919c001520 con 0x7f91ac0721c0 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:06.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91aaffd640 10 monclient: dump: 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.349+0000 7f91b292b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.350+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f919c001680 con 0x7f91ac0721c0 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.350+0000 7f91aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.350+0000 7f91b292b640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:06.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.350+0000 7f91b292b640 4 set_mon_vals no callback set 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.351+0000 7f91aaffd640 1 -- 172.21.15.157:0/2621612562 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f919c001a50 con 0x7f91ac0721c0 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.351+0000 7f91aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.351+0000 7f91aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.351+0000 7f91aaffd640 10 monclient: dump: 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:06.332 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:06.333 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.351+0000 7f91b292b640 10 monclient: shutdown 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.351+0000 7f91b292b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.351+0000 7f91b292b640 1 -- 172.21.15.157:0/2621612562 >> v1:172.21.15.157:6789/0 conn(0x7f91ac0721c0 legacy=0x7f91ac13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.352+0000 7f91b292b640 1 -- 172.21.15.157:0/2621612562 shutdown_connections 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.352+0000 7f91b292b640 1 -- 172.21.15.157:0/2621612562 >> 172.21.15.157:0/2621612562 conn(0x7f91ac06b660 msgr2=0x7f91ac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.352+0000 7f91b292b640 1 -- 172.21.15.157:0/2621612562 shutdown_connections 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.352+0000 7f91b292b640 1 -- 172.21.15.157:0/2621612562 wait complete. 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.353+0000 7f91b292b640 10 monclient: build_initial_monmap 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.353+0000 7f91b292b640 10 monclient: monmap: 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:06.334 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.353+0000 7f91b292b640 1 librados: starting msgr at 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.353+0000 7f91b292b640 1 librados: starting objecter 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.353+0000 7f91b292b640 1 Processor -- start 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.353+0000 7f91b292b640 1 -- start start 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.353+0000 7f91b292b640 1 librados: setting wanted keys 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.353+0000 7f91b292b640 1 librados: calling monclient init 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.353+0000 7f91b292b640 10 monclient: init 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.354+0000 7f91b292b640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.354+0000 7f91b292b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.354+0000 7f91b292b640 10 monclient(hunting): picked mon.noname-a con 0x7f91ac071de0 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.354+0000 7f91b292b640 10 monclient(hunting): picked mon.noname-b con 0x7f91ac145ae0 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:06.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.354+0000 7f91b292b640 10 monclient(hunting): picked mon.noname-c con 0x7f91ac138270 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:06.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.354+0000 7f91b292b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac06fba0 con 0x7f91ac071de0 2024-03-29T11:12:06.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.354+0000 7f91b292b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac070ba0 con 0x7f91ac145ae0 2024-03-29T11:12:06.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.354+0000 7f91b292b640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac070e80 con 0x7f91ac138270 2024-03-29T11:12:06.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.354+0000 7f91b292b640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:06.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.354+0000 7f91b292b640 10 monclient(hunting): authenticate will time out at 938.551626s 2024-03-29T11:12:06.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.355+0000 7f91ab7fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f91ac145ae0 0x7f91ac137b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:38240/0 (socket says 172.21.15.157:38240) 2024-03-29T11:12:06.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.355+0000 7f91ab7fe640 1 -- 172.21.15.157:0/2151954735 learned_addr learned my addr 172.21.15.157:0/2151954735 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:06.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.355+0000 7f91a8ff9640 10 client.?.objecter ms_handle_connect 0x7f91ac145ae0 2024-03-29T11:12:06.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.355+0000 7f91a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:06.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.356+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357152791 0 0) 0x7f91ac070ba0 con 0x7f91ac145ae0 2024-03-29T11:12:06.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.356+0000 7f91a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:06.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.356+0000 7f91a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.356+0000 7f91a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.356+0000 7f91a8ff9640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.356+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f918c004eb0 con 0x7f91ac145ae0 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.357+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3214368976 0 0) 0x7f918c004eb0 con 0x7f91ac145ae0 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.358+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91ac070ba0 con 0x7f91ac145ae0 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.358+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f919c001990 con 0x7f91ac145ae0 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.358+0000 7f91a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.358+0000 7f91a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.358+0000 7f91a8ff9640 10 monclient(hunting): dump: 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:06.342 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.358+0000 7f91a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 10 client.?.objecter ms_handle_connect 0x7f91ac138270 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2633241048 0 0) 0x7f91ac070ba0 con 0x7f91ac145ae0 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 >> v1:172.21.15.157:6790/0 conn(0x7f91ac138270 legacy=0x7f91ac1e61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 >> v1:172.21.15.157:6789/0 conn(0x7f91ac071de0 legacy=0x7f91ac071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:12:06.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 1 monclient: found mon.b 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac06f830 con 0x7f91ac145ae0 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91b292b640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91b292b640 10 monclient: _renew_subs 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91b292b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91b292b640 1 -- 172.21.15.157:0/2151954735 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ac1360b0 con 0x7f91ac145ae0 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91b292b640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91b292b640 10 monclient: _renew_subs 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91b292b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91b292b640 1 -- 172.21.15.157:0/2151954735 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac1e78e0 con 0x7f91ac145ae0 2024-03-29T11:12:06.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.360+0000 7f91b292b640 1 librados: init done 2024-03-29T11:12:06.346 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.361+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f919c0038a0 con 0x7f91ac145ae0 2024-03-29T11:12:06.346 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.361+0000 7f91a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.361+0000 7f91aa7fc640 4 set_mon_vals no callback set 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.361+0000 7f91b292b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.362+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f919c0010e0 con 0x7f91ac145ae0 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.362+0000 7f91a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.362+0000 7f91a8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.362+0000 7f91a8ff9640 10 monclient: dump: 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:06.347 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:06.348 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:06.348 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:06.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.363+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f919c003cb0 con 0x7f91ac145ae0 2024-03-29T11:12:06.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.364+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f919c084be0 con 0x7f91ac145ae0 2024-03-29T11:12:06.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.364+0000 7f91a8ff9640 10 client.4255.objecter ms_dispatch 0x7f91ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:06.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.364+0000 7f91a8ff9640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:06.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.364+0000 7f91a8ff9640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:06.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.364+0000 7f91a8ff9640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T11:12:06.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.364+0000 7f91b292b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:06.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.364+0000 7f91b292b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:06.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.364+0000 7f91b292b640 1 -- 172.21.15.157:0/2151954735 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9190005080 con 0x7f91ac145ae0 2024-03-29T11:12:06.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.364+0000 7f91abfff640 10 monclient: get_auth_request con 0x7f918c06dc60 auth_method 0 2024-03-29T11:12:06.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.365+0000 7f91a8ff9640 10 client.4255.objecter ms_handle_connect 0x7f918c06dc60 2024-03-29T11:12:06.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.375+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f919c0504c0 con 0x7f91ac145ae0 2024-03-29T11:12:06.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.375+0000 7f91a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:06.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.375+0000 7f91a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:06.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.698+0000 7f91b292b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:06.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.698+0000 7f91b292b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:06.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.699+0000 7f91b292b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:06.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.699+0000 7f91b292b640 1 -- 172.21.15.157:0/2151954735 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9190005330 con 0x7f91ac145ae0 2024-03-29T11:12:06.678 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.700+0000 7f91a8ff9640 1 -- 172.21.15.157:0/2151954735 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f919c0523f0 con 0x7f91ac145ae0 2024-03-29T11:12:06.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.700+0000 7f91a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:06.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.700+0000 7f91a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:06.679 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:12:06.679 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:06.682 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.704+0000 7f91b292b640 10 librados: watch_flush enter 2024-03-29T11:12:06.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.705+0000 7f91b292b640 10 librados: watch_flush exit 2024-03-29T11:12:06.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.705+0000 7f91b292b640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T11:12:06.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.705+0000 7f91b292b640 10 client.4255.objecter successfully canceled tick 2024-03-29T11:12:06.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.705+0000 7f91b292b640 1 -- 172.21.15.157:0/2151954735 >> v1:172.21.15.157:6816/1747058247 conn(0x7f918c06dc60 legacy=0x7f918c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.705+0000 7f91b292b640 10 monclient: shutdown 2024-03-29T11:12:06.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.705+0000 7f91b292b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:06.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.705+0000 7f91b292b640 1 -- 172.21.15.157:0/2151954735 >> v1:172.21.15.195:6789/0 conn(0x7f91ac145ae0 legacy=0x7f91ac137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:06.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.706+0000 7f91b292b640 1 -- 172.21.15.157:0/2151954735 shutdown_connections 2024-03-29T11:12:06.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.706+0000 7f91b292b640 1 -- 172.21.15.157:0/2151954735 >> 172.21.15.157:0/2151954735 conn(0x7f91ac06b660 msgr2=0x7f91ac06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:06.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.706+0000 7f91b292b640 1 -- 172.21.15.157:0/2151954735 shutdown_connections 2024-03-29T11:12:06.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.706+0000 7f91b292b640 1 -- 172.21.15.157:0/2151954735 wait complete. 2024-03-29T11:12:06.685 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:06.707+0000 7f91b292b640 1 librados: shutdown 2024-03-29T11:12:06.704 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:13.511 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:13.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.687+0000 7f3da759e640 10 monclient: get_monmap_and_config 2024-03-29T11:12:13.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.687+0000 7f3da759e640 10 monclient: build_initial_monmap 2024-03-29T11:12:13.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.687+0000 7f3da759e640 10 monclient: monmap: 2024-03-29T11:12:13.667 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:13.667 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:13.667 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:12:13.688457+0000 2024-03-29T11:12:13.668 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:12:13.688457+0000 2024-03-29T11:12:13.668 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:13.668 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:13.668 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:13.668 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:12:13.668 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:12:13.668 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:13.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.690+0000 7f3da759e640 1 Processor -- start 2024-03-29T11:12:13.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.690+0000 7f3da759e640 1 -- start start 2024-03-29T11:12:13.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.690+0000 7f3da759e640 10 monclient: init 2024-03-29T11:12:13.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.691+0000 7f3da759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:13.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.691+0000 7f3da759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:13.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.691+0000 7f3da759e640 10 monclient(hunting): picked mon.noname-b con 0x7f3da0034850 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:13.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.691+0000 7f3da759e640 10 monclient(hunting): picked mon.noname-a con 0x7f3da002a640 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:13.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.691+0000 7f3da759e640 10 monclient(hunting): picked mon.noname-c con 0x7f3da00327d0 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:13.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.691+0000 7f3da759e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da002c820 con 0x7f3da002a640 2024-03-29T11:12:13.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.691+0000 7f3da759e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da002a120 con 0x7f3da00327d0 2024-03-29T11:12:13.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.691+0000 7f3da759e640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da00c75b0 con 0x7f3da0034850 2024-03-29T11:12:13.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.691+0000 7f3da759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:13.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.691+0000 7f3da759e640 10 monclient(hunting): authenticate will time out at 945.888588s 2024-03-29T11:12:13.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.695+0000 7f3da659c640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f3da0034850 0x7f3da00c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:59017/0 (socket says 172.21.15.157:59017) 2024-03-29T11:12:13.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.695+0000 7f3da659c640 1 -- 172.21.15.157:0/738734805 learned_addr learned my addr 172.21.15.157:0/738734805 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:13.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.695+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046451168 0 0) 0x7f3da00c75b0 con 0x7f3da0034850 2024-03-29T11:12:13.678 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.695+0000 7f3da559a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:13.678 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.695+0000 7f3da559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:13.678 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.695+0000 7f3da559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:13.678 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.695+0000 7f3da559a640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:12:13.678 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.695+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d90003520 con 0x7f3da0034850 2024-03-29T11:12:13.678 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472399914 0 0) 0x7f3da002a120 con 0x7f3da00327d0 2024-03-29T11:12:13.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:13.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:13.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:13.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:12:13.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da00c75b0 con 0x7f3da00327d0 2024-03-29T11:12:13.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722666731 0 0) 0x7f3da002c820 con 0x7f3da002a640 2024-03-29T11:12:13.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:13.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:13.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:13.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:12:13.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da002a120 con 0x7f3da002a640 2024-03-29T11:12:13.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2497918465 0 0) 0x7f3d90003520 con 0x7f3da0034850 2024-03-29T11:12:13.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 >> v1:172.21.15.195:6789/0 conn(0x7f3da00327d0 legacy=0x7f3da00cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 >> v1:172.21.15.157:6789/0 conn(0x7f3da002a640 legacy=0x7f3da00320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:13.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 1 monclient: found mon.noname-b 2024-03-29T11:12:13.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:13.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:12:13.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da0033110 con 0x7f3da0034850 2024-03-29T11:12:13.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient: _finish_auth 0 2024-03-29T11:12:13.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:13.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:13.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3d9c001690 con 0x7f3da0034850 2024-03-29T11:12:13.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:13.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:13.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.696+0000 7f3da559a640 10 monclient: dump: 2024-03-29T11:12:13.682 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:13.682 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:13.682 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:13.682 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:13.682 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:13.682 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:13.682 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.697+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d9c0018b0 con 0x7f3da0034850 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.697+0000 7f3da759e640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.697+0000 7f3da759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.697+0000 7f3da559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.697+0000 7f3da559a640 1 -- 172.21.15.157:0/738734805 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3d9c001c40 con 0x7f3da0034850 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.697+0000 7f3da559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.697+0000 7f3da559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.697+0000 7f3da559a640 10 monclient: dump: 2024-03-29T11:12:13.683 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.697+0000 7f3da759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.697+0000 7f3da759e640 4 set_mon_vals no callback set 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.698+0000 7f3da759e640 10 monclient: shutdown 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.698+0000 7f3da759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.698+0000 7f3da759e640 1 -- 172.21.15.157:0/738734805 >> v1:172.21.15.157:6790/0 conn(0x7f3da0034850 legacy=0x7f3da00c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.698+0000 7f3da759e640 1 -- 172.21.15.157:0/738734805 shutdown_connections 2024-03-29T11:12:13.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.698+0000 7f3da759e640 1 -- 172.21.15.157:0/738734805 >> 172.21.15.157:0/738734805 conn(0x7f3da00058f0 msgr2=0x7f3da0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:13.695 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.699+0000 7f3da759e640 1 -- 172.21.15.157:0/738734805 shutdown_connections 2024-03-29T11:12:13.695 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.700+0000 7f3da759e640 1 -- 172.21.15.157:0/738734805 wait complete. 2024-03-29T11:12:13.695 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.701+0000 7f3da759e640 10 monclient: build_initial_monmap 2024-03-29T11:12:13.696 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.701+0000 7f3da759e640 10 monclient: monmap: 2024-03-29T11:12:13.696 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:13.696 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:13.696 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:12:13.696 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:12:13.696 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:13.696 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:13.696 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:13.697 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:12:13.697 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:12:13.697 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:13.697 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.701+0000 7f3da759e640 1 librados: starting msgr at 2024-03-29T11:12:13.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.701+0000 7f3da759e640 1 librados: starting objecter 2024-03-29T11:12:13.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.701+0000 7f3da759e640 1 Processor -- start 2024-03-29T11:12:13.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 1 -- start start 2024-03-29T11:12:13.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 1 librados: setting wanted keys 2024-03-29T11:12:13.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 1 librados: calling monclient init 2024-03-29T11:12:13.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 10 monclient: init 2024-03-29T11:12:13.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:13.699 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:13.699 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 10 monclient(hunting): picked mon.noname-b con 0x7f3da002a640 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:13.699 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 10 monclient(hunting): picked mon.noname-a con 0x7f3da00327d0 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:13.699 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 10 monclient(hunting): picked mon.noname-c con 0x7f3da0034850 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:13.699 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da002dcf0 con 0x7f3da00327d0 2024-03-29T11:12:13.699 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da002dfb0 con 0x7f3da002a640 2024-03-29T11:12:13.699 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da002d2c0 con 0x7f3da0034850 2024-03-29T11:12:13.699 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:13.699 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.702+0000 7f3da759e640 10 monclient(hunting): authenticate will time out at 945.899856s 2024-03-29T11:12:13.699 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.708+0000 7f3da6d9d640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f3da0034850 0x7f3da016c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:59089/0 (socket says 172.21.15.157:59089) 2024-03-29T11:12:13.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.708+0000 7f3da6d9d640 1 -- 172.21.15.157:0/2178901799 learned_addr learned my addr 172.21.15.157:0/2178901799 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:13.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.708+0000 7f3d96ffd640 10 client.?.objecter ms_handle_connect 0x7f3da0034850 2024-03-29T11:12:13.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.708+0000 7f3d96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:13.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.708+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082368613 0 0) 0x7f3da002d2c0 con 0x7f3da0034850 2024-03-29T11:12:13.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.708+0000 7f3d96ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:13.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.708+0000 7f3d96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:13.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.708+0000 7f3d96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:13.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.708+0000 7f3d96ffd640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.708+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d88004eb0 con 0x7f3da0034850 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.709+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4145134341 0 0) 0x7f3d88004eb0 con 0x7f3da0034850 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.709+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3da002d2c0 con 0x7f3da0034850 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.709+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3da804fdd0 con 0x7f3da0034850 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.709+0000 7f3d96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.709+0000 7f3d96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.709+0000 7f3d96ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:13.703 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.709+0000 7f3d96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.709+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 272155812 0 0) 0x7f3da002d2c0 con 0x7f3da0034850 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.709+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 >> v1:172.21.15.195:6789/0 conn(0x7f3da002a640 legacy=0x7f3da0164180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.710+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 >> v1:172.21.15.157:6789/0 conn(0x7f3da00327d0 legacy=0x7f3da0030fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.710+0000 7f3d96ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.710+0000 7f3d96ffd640 1 monclient: found mon.c 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.710+0000 7f3d96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.710+0000 7f3d96ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.710+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da002d450 con 0x7f3da0034850 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.710+0000 7f3d96ffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.710+0000 7f3d96ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:13.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.710+0000 7f3d96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.711+0000 7f3da759e640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.711+0000 7f3da759e640 10 monclient: _renew_subs 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.711+0000 7f3da759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.711+0000 7f3da759e640 1 -- 172.21.15.157:0/2178901799 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da002d7a0 con 0x7f3da0034850 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.711+0000 7f3da759e640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.711+0000 7f3da759e640 10 monclient: _renew_subs 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.711+0000 7f3da759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.711+0000 7f3da759e640 1 -- 172.21.15.157:0/2178901799 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da00d0c40 con 0x7f3da0034850 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.711+0000 7f3da759e640 1 librados: init done 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.712+0000 7f3d96ffd640 10 client.4262.objecter ms_handle_connect 0x7f3da00327d0 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.712+0000 7f3d96ffd640 10 client.4262.objecter resend_mon_ops 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.712+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3da804f400 con 0x7f3da0034850 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.712+0000 7f3d96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.712+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3da8075f10 con 0x7f3da0034850 2024-03-29T11:12:13.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.713+0000 7f3d96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.713+0000 7f3d96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.713+0000 7f3d96ffd640 10 monclient: dump: 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.713+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f3da8067bf0 con 0x7f3da0034850 2024-03-29T11:12:13.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.714+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f3da80ff690 con 0x7f3da0034850 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.714+0000 7f3d96ffd640 10 client.4262.objecter ms_dispatch 0x7f3da8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.714+0000 7f3d96ffd640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.714+0000 7f3d96ffd640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.714+0000 7f3d96ffd640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.714+0000 7f3da4d99640 4 set_mon_vals no callback set 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.715+0000 7f3da759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.715+0000 7f3da759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.715+0000 7f3da759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.715+0000 7f3da759e640 1 -- 172.21.15.157:0/2178901799 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c005080 con 0x7f3da0034850 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.716+0000 7f3da659c640 10 monclient: get_auth_request con 0x7f3d8806dad0 auth_method 0 2024-03-29T11:12:13.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.716+0000 7f3d96ffd640 10 client.4262.objecter ms_handle_connect 0x7f3d8806dad0 2024-03-29T11:12:13.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.724+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3da80cb0b0 con 0x7f3da0034850 2024-03-29T11:12:13.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.724+0000 7f3d96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:13.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:13.724+0000 7f3d96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:14.026 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.048+0000 7f3da759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:14.026 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.048+0000 7f3da759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:14.026 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.048+0000 7f3da759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:14.026 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.048+0000 7f3da759e640 1 -- 172.21.15.157:0/2178901799 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d8c005330 con 0x7f3da0034850 2024-03-29T11:12:14.027 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.049+0000 7f3d96ffd640 1 -- 172.21.15.157:0/2178901799 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f3da80cb0b0 con 0x7f3da0034850 2024-03-29T11:12:14.027 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.049+0000 7f3d96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:14.027 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.049+0000 7f3d96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:14.027 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:12:14.027 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:14.030 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.052+0000 7f3da759e640 10 librados: watch_flush enter 2024-03-29T11:12:14.030 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.052+0000 7f3da759e640 10 librados: watch_flush exit 2024-03-29T11:12:14.030 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.052+0000 7f3da759e640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:12:14.030 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.052+0000 7f3da759e640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:12:14.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.052+0000 7f3da759e640 1 -- 172.21.15.157:0/2178901799 >> v1:172.21.15.157:6816/1747058247 conn(0x7f3d8806dad0 legacy=0x7f3d8806ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:14.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.052+0000 7f3da759e640 10 monclient: shutdown 2024-03-29T11:12:14.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.052+0000 7f3da759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:14.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.052+0000 7f3da759e640 1 -- 172.21.15.157:0/2178901799 >> v1:172.21.15.157:6790/0 conn(0x7f3da0034850 legacy=0x7f3da016c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:14.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.053+0000 7f3da759e640 1 -- 172.21.15.157:0/2178901799 shutdown_connections 2024-03-29T11:12:14.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.053+0000 7f3da759e640 1 -- 172.21.15.157:0/2178901799 >> 172.21.15.157:0/2178901799 conn(0x7f3da00058f0 msgr2=0x7f3da00043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:14.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.053+0000 7f3da759e640 1 -- 172.21.15.157:0/2178901799 shutdown_connections 2024-03-29T11:12:14.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.053+0000 7f3da759e640 1 -- 172.21.15.157:0/2178901799 wait complete. 2024-03-29T11:12:14.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:14.053+0000 7f3da759e640 1 librados: shutdown 2024-03-29T11:12:14.051 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:20.859 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:21.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.023+0000 7f6ee644a640 10 monclient: get_monmap_and_config 2024-03-29T11:12:21.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.023+0000 7f6ee644a640 10 monclient: build_initial_monmap 2024-03-29T11:12:21.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.023+0000 7f6ee644a640 10 monclient: monmap: 2024-03-29T11:12:21.004 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:21.004 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:21.005 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:12:21.024453+0000 2024-03-29T11:12:21.005 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:12:21.024453+0000 2024-03-29T11:12:21.005 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:21.005 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:21.005 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:21.005 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:12:21.005 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:12:21.005 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:21.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.026+0000 7f6ee644a640 1 Processor -- start 2024-03-29T11:12:21.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.026+0000 7f6ee644a640 1 -- start start 2024-03-29T11:12:21.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.026+0000 7f6ee644a640 10 monclient: init 2024-03-29T11:12:21.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.027+0000 7f6ee644a640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:21.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.027+0000 7f6ee644a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:21.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.030+0000 7f6ee644a640 10 monclient(hunting): picked mon.noname-c con 0x7f6ee00b4100 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:21.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.030+0000 7f6ee644a640 10 monclient(hunting): picked mon.noname-a con 0x7f6ee013ee90 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:21.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.030+0000 7f6edffff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6ee00b4100 0x7f6ee00adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:43299/0 (socket says 172.21.15.157:43299) 2024-03-29T11:12:21.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.031+0000 7f6ee644a640 10 monclient(hunting): picked mon.noname-b con 0x7f6ee00af900 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:21.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.031+0000 7f6ee644a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee00afd10 con 0x7f6ee013ee90 2024-03-29T11:12:21.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.031+0000 7f6ee644a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee00af6f0 con 0x7f6ee00b4100 2024-03-29T11:12:21.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.031+0000 7f6ee644a640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee00aac80 con 0x7f6ee00af900 2024-03-29T11:12:21.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.031+0000 7f6ee644a640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:21.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.031+0000 7f6ee644a640 10 monclient(hunting): authenticate will time out at 953.228085s 2024-03-29T11:12:21.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.031+0000 7f6edffff640 1 -- 172.21.15.157:0/2100396425 learned_addr learned my addr 172.21.15.157:0/2100396425 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:21.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 1 -- 172.21.15.157:0/2100396425 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539936491 0 0) 0x7f6ee00af6f0 con 0x7f6ee00b4100 2024-03-29T11:12:21.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:21.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:21.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:21.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:12:21.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 1 -- 172.21.15.157:0/2100396425 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ec8005730 con 0x7f6ee00b4100 2024-03-29T11:12:21.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 1 -- 172.21.15.157:0/2100396425 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 114700556 0 0) 0x7f6ec8005730 con 0x7f6ee00b4100 2024-03-29T11:12:21.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 1 -- 172.21.15.157:0/2100396425 >> v1:172.21.15.157:6790/0 conn(0x7f6ee00af900 legacy=0x7f6ee014ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:21.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 1 -- 172.21.15.157:0/2100396425 >> v1:172.21.15.157:6789/0 conn(0x7f6ee013ee90 legacy=0x7f6ee013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:21.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:21.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 1 monclient: found mon.noname-c 2024-03-29T11:12:21.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:21.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:12:21.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 1 -- 172.21.15.157:0/2100396425 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee00aaf60 con 0x7f6ee00b4100 2024-03-29T11:12:21.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:21.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:21.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:21.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 1 -- 172.21.15.157:0/2100396425 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6ed0001690 con 0x7f6ee00b4100 2024-03-29T11:12:21.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.034+0000 7f6edeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:21.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.035+0000 7f6edeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:21.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.035+0000 7f6edeffd640 10 monclient: dump: 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.035+0000 7f6ee644a640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.035+0000 7f6ee644a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.035+0000 7f6edeffd640 1 -- 172.21.15.157:0/2100396425 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ed0001860 con 0x7f6ee00b4100 2024-03-29T11:12:21.018 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.035+0000 7f6edeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.035+0000 7f6ee644a640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.035+0000 7f6ee644a640 4 set_mon_vals no callback set 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.036+0000 7f6ee644a640 10 monclient: shutdown 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.036+0000 7f6ee644a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.036+0000 7f6ee644a640 1 -- 172.21.15.157:0/2100396425 >> v1:172.21.15.195:6789/0 conn(0x7f6ee00b4100 legacy=0x7f6ee00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.036+0000 7f6ee644a640 1 -- 172.21.15.157:0/2100396425 shutdown_connections 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.036+0000 7f6ee644a640 1 -- 172.21.15.157:0/2100396425 >> 172.21.15.157:0/2100396425 conn(0x7f6ee00b35e0 msgr2=0x7f6ee00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.037+0000 7f6ee644a640 1 -- 172.21.15.157:0/2100396425 shutdown_connections 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.037+0000 7f6ee644a640 1 -- 172.21.15.157:0/2100396425 wait complete. 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.038+0000 7f6ee644a640 10 monclient: build_initial_monmap 2024-03-29T11:12:21.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.038+0000 7f6ee644a640 10 monclient: monmap: 2024-03-29T11:12:21.020 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:21.020 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:21.020 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:12:21.020 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:12:21.020 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:21.020 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:21.020 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:21.020 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:12:21.020 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:12:21.020 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:21.021 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.039+0000 7f6ee644a640 1 librados: starting msgr at 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.039+0000 7f6ee644a640 1 librados: starting objecter 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.039+0000 7f6ee644a640 1 Processor -- start 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.039+0000 7f6ee644a640 1 -- start start 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.039+0000 7f6ee644a640 1 librados: setting wanted keys 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.039+0000 7f6ee644a640 1 librados: calling monclient init 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.039+0000 7f6ee644a640 10 monclient: init 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.040+0000 7f6ee644a640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.040+0000 7f6ee644a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.042+0000 7f6ee644a640 10 monclient(hunting): picked mon.noname-a con 0x7f6ee00af900 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.042+0000 7f6ee644a640 10 monclient(hunting): picked mon.noname-c con 0x7f6ee00b4100 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.042+0000 7f6ee644a640 10 monclient(hunting): picked mon.noname-b con 0x7f6ee013ee90 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.042+0000 7f6ee644a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee011c5f0 con 0x7f6ee00af900 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.042+0000 7f6ee644a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee011c1a0 con 0x7f6ee013ee90 2024-03-29T11:12:21.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.042+0000 7f6ee644a640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee011c330 con 0x7f6ee00b4100 2024-03-29T11:12:21.023 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.042+0000 7f6ee644a640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:21.023 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.042+0000 7f6ee644a640 10 monclient(hunting): authenticate will time out at 953.239640s 2024-03-29T11:12:21.024 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.046+0000 7f6edffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6ee00af900 0x7f6ee01392a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:41943/0 (socket says 172.21.15.157:41943) 2024-03-29T11:12:21.024 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.046+0000 7f6edffff640 1 -- 172.21.15.157:0/2491475504 learned_addr learned my addr 172.21.15.157:0/2491475504 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:21.027 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 client.?.objecter ms_handle_connect 0x7f6ee00af900 2024-03-29T11:12:21.028 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:21.028 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 client.?.objecter ms_handle_connect 0x7f6ee00b4100 2024-03-29T11:12:21.028 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:21.028 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902925665 0 0) 0x7f6ee011c5f0 con 0x7f6ee00af900 2024-03-29T11:12:21.028 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:21.028 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:21.028 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:21.028 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ecc004eb0 con 0x7f6ee00af900 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154860358 0 0) 0x7f6ee011c330 con 0x7f6ee00b4100 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee011c5f0 con 0x7f6ee00b4100 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 614551576 0 0) 0x7f6ee011c5f0 con 0x7f6ee00b4100 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ee011c330 con 0x7f6ee00b4100 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6ed4001640 con 0x7f6ee00b4100 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:21.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): dump: 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2101683695 0 0) 0x7f6ecc004eb0 con 0x7f6ee00af900 2024-03-29T11:12:21.030 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ee011c5f0 con 0x7f6ee00af900 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6ed0004ae0 con 0x7f6ee00af900 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient(hunting): dump: 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:21.031 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2535233341 0 0) 0x7f6ee011c5f0 con 0x7f6ee00af900 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 >> v1:172.21.15.157:6790/0 conn(0x7f6ee00b4100 legacy=0x7f6ee011cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 >> v1:172.21.15.195:6789/0 conn(0x7f6ee013ee90 legacy=0x7f6ee01e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 monclient: found mon.a 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:12:21.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.047+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee0139f90 con 0x7f6ee00af900 2024-03-29T11:12:21.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.048+0000 7f6edcff9640 10 monclient: _finish_auth 0 2024-03-29T11:12:21.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.048+0000 7f6edcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:12:21.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.048+0000 7f6edcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:21.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.048+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ed0004ef0 con 0x7f6ee00af900 2024-03-29T11:12:21.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.048+0000 7f6edcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:21.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.048+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6ed0005340 con 0x7f6ee00af900 2024-03-29T11:12:21.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.048+0000 7f6edcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:21.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.048+0000 7f6edcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:21.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.048+0000 7f6edcff9640 10 monclient: dump: 2024-03-29T11:12:21.033 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:21.034 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:21.034 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:21.034 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:21.034 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:21.034 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:21.034 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:21.034 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:21.034 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:21.034 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:21.038 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.049+0000 7f6ede7fc640 4 set_mon_vals no callback set 2024-03-29T11:12:21.039 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.051+0000 7f6ee644a640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:12:21.039 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.051+0000 7f6ee644a640 10 monclient: _renew_subs 2024-03-29T11:12:21.039 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.051+0000 7f6ee644a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:12:21.039 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.051+0000 7f6ee644a640 1 -- 172.21.15.157:0/2491475504 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee01d7dc0 con 0x7f6ee00af900 2024-03-29T11:12:21.039 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.051+0000 7f6ee644a640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.051+0000 7f6ee644a640 10 monclient: _renew_subs 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.051+0000 7f6ee644a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.051+0000 7f6ee644a640 1 -- 172.21.15.157:0/2491475504 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee01d7560 con 0x7f6ee00af900 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.051+0000 7f6ee644a640 1 librados: init done 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.053+0000 7f6ee644a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.053+0000 7f6ee644a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.053+0000 7f6ee644a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.053+0000 7f6ee644a640 1 -- 172.21.15.157:0/2491475504 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee011c780 con 0x7f6ee00af900 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.054+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f6ed0001b10 con 0x7f6ee00af900 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.055+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f6ed0084d30 con 0x7f6ee00af900 2024-03-29T11:12:21.040 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.055+0000 7f6edcff9640 10 client.4260.objecter ms_dispatch 0x7f6ee0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:21.041 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.055+0000 7f6edcff9640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:21.041 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.055+0000 7f6edcff9640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:21.041 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.055+0000 7f6edf7fe640 10 monclient: get_auth_request con 0x7f6ecc06df00 auth_method 0 2024-03-29T11:12:21.041 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.055+0000 7f6edcff9640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T11:12:21.041 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.056+0000 7f6edcff9640 10 client.4260.objecter ms_handle_connect 0x7f6ecc06df00 2024-03-29T11:12:21.042 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.064+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ed0050610 con 0x7f6ee00af900 2024-03-29T11:12:21.042 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.064+0000 7f6edcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:21.042 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.064+0000 7f6edcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:21.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.378+0000 7f6ee644a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:21.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.379+0000 7f6ee644a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:21.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.379+0000 7f6ee644a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:12:21.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.379+0000 7f6ee644a640 1 -- 172.21.15.157:0/2491475504 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ee006dfd0 con 0x7f6ee00af900 2024-03-29T11:12:21.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.379+0000 7f6edcff9640 1 -- 172.21.15.157:0/2491475504 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f6ed0052540 con 0x7f6ee00af900 2024-03-29T11:12:21.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.379+0000 7f6edcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:21.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.379+0000 7f6edcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:21.358 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:12:21.358 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:21.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.383+0000 7f6ee644a640 10 librados: watch_flush enter 2024-03-29T11:12:21.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.383+0000 7f6ee644a640 10 librados: watch_flush exit 2024-03-29T11:12:21.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.383+0000 7f6ee644a640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T11:12:21.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.383+0000 7f6ee644a640 10 client.4260.objecter successfully canceled tick 2024-03-29T11:12:21.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.383+0000 7f6ee644a640 1 -- 172.21.15.157:0/2491475504 >> v1:172.21.15.157:6816/1747058247 conn(0x7f6ecc06df00 legacy=0x7f6ecc0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:21.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.383+0000 7f6ee644a640 10 monclient: shutdown 2024-03-29T11:12:21.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.383+0000 7f6ee644a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:21.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.383+0000 7f6ee644a640 1 -- 172.21.15.157:0/2491475504 >> v1:172.21.15.157:6789/0 conn(0x7f6ee00af900 legacy=0x7f6ee01392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:21.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.385+0000 7f6ee644a640 1 -- 172.21.15.157:0/2491475504 shutdown_connections 2024-03-29T11:12:21.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.385+0000 7f6ee644a640 1 -- 172.21.15.157:0/2491475504 >> 172.21.15.157:0/2491475504 conn(0x7f6ee00b35e0 msgr2=0x7f6ee013ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:21.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.387+0000 7f6ee644a640 1 -- 172.21.15.157:0/2491475504 shutdown_connections 2024-03-29T11:12:21.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.387+0000 7f6ee644a640 1 -- 172.21.15.157:0/2491475504 wait complete. 2024-03-29T11:12:21.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:21.387+0000 7f6ee644a640 1 librados: shutdown 2024-03-29T11:12:21.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:28.192 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:28.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.368+0000 7f24f4825640 10 monclient: get_monmap_and_config 2024-03-29T11:12:28.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.368+0000 7f24f4825640 10 monclient: build_initial_monmap 2024-03-29T11:12:28.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.368+0000 7f24f4825640 10 monclient: monmap: 2024-03-29T11:12:28.350 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:28.350 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:28.350 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:12:28.369371+0000 2024-03-29T11:12:28.350 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:12:28.369371+0000 2024-03-29T11:12:28.350 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:28.350 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:28.350 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:28.350 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:12:28.351 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:12:28.351 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:28.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.370+0000 7f24f4825640 1 Processor -- start 2024-03-29T11:12:28.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.370+0000 7f24f4825640 1 -- start start 2024-03-29T11:12:28.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.370+0000 7f24f4825640 10 monclient: init 2024-03-29T11:12:28.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.371+0000 7f24f4825640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:28.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.371+0000 7f24f4825640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:28.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.371+0000 7f24f4825640 10 monclient(hunting): picked mon.noname-c con 0x7f24f00aabc0 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:28.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.371+0000 7f24f4825640 10 monclient(hunting): picked mon.noname-b con 0x7f24f00ad0b0 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:28.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.371+0000 7f24f4825640 10 monclient(hunting): picked mon.noname-a con 0x7f24f00ad530 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:28.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.371+0000 7f24f4825640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f00ade20 con 0x7f24f00ad530 2024-03-29T11:12:28.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.371+0000 7f24f4825640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f00aafa0 con 0x7f24f00aabc0 2024-03-29T11:12:28.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.371+0000 7f24f4825640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f00ac580 con 0x7f24f00ad0b0 2024-03-29T11:12:28.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.371+0000 7f24f4825640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:28.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.371+0000 7f24f4825640 10 monclient(hunting): authenticate will time out at 960.568690s 2024-03-29T11:12:28.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.375+0000 7f24ee59c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f24f00aabc0 0x7f24f00ac9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:59851/0 (socket says 172.21.15.157:59851) 2024-03-29T11:12:28.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.375+0000 7f24ee59c640 1 -- 172.21.15.157:0/1477889005 learned_addr learned my addr 172.21.15.157:0/1477889005 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:28.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791853078 0 0) 0x7f24f00ac580 con 0x7f24f00ad0b0 2024-03-29T11:12:28.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:28.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:28.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:28.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:12:28.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d8003520 con 0x7f24f00ad0b0 2024-03-29T11:12:28.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612501984 0 0) 0x7f24f00ade20 con 0x7f24f00ad530 2024-03-29T11:12:28.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:28.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:28.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:28.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.378+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f00ac580 con 0x7f24f00ad530 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2743359581 0 0) 0x7f24f00ac580 con 0x7f24f00ad530 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 >> v1:172.21.15.157:6790/0 conn(0x7f24f00ad0b0 legacy=0x7f24f00ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 >> v1:172.21.15.195:6789/0 conn(0x7f24f00aabc0 legacy=0x7f24f00ac9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 1 monclient: found mon.noname-a 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f0155ad0 con 0x7f24f00ad530 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 10 monclient: _finish_auth 0 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f24e0001670 con 0x7f24f00ad530 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:28.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24ed59a640 10 monclient: dump: 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24f4825640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.380+0000 7f24f4825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.383+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24e00017d0 con 0x7f24f00ad530 2024-03-29T11:12:28.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.383+0000 7f24ed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.383+0000 7f24ed59a640 1 -- 172.21.15.157:0/1477889005 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f24e0001be0 con 0x7f24f00ad530 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.383+0000 7f24ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.383+0000 7f24ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.383+0000 7f24ed59a640 10 monclient: dump: 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:28.366 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:28.367 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:28.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.384+0000 7f24f4825640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:28.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.384+0000 7f24f4825640 4 set_mon_vals no callback set 2024-03-29T11:12:28.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.385+0000 7f24f4825640 10 monclient: shutdown 2024-03-29T11:12:28.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.385+0000 7f24f4825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:28.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.385+0000 7f24f4825640 1 -- 172.21.15.157:0/1477889005 >> v1:172.21.15.157:6789/0 conn(0x7f24f00ad530 legacy=0x7f24f015bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.385+0000 7f24f4825640 1 -- 172.21.15.157:0/1477889005 shutdown_connections 2024-03-29T11:12:28.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.385+0000 7f24f4825640 1 -- 172.21.15.157:0/1477889005 >> 172.21.15.157:0/1477889005 conn(0x7f24f00b3c80 msgr2=0x7f24f00b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:28.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.385+0000 7f24f4825640 1 -- 172.21.15.157:0/1477889005 shutdown_connections 2024-03-29T11:12:28.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.385+0000 7f24f4825640 1 -- 172.21.15.157:0/1477889005 wait complete. 2024-03-29T11:12:28.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.386+0000 7f24f4825640 10 monclient: build_initial_monmap 2024-03-29T11:12:28.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.386+0000 7f24f4825640 10 monclient: monmap: 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.387+0000 7f24f4825640 1 librados: starting msgr at 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.387+0000 7f24f4825640 1 librados: starting objecter 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.387+0000 7f24f4825640 1 Processor -- start 2024-03-29T11:12:28.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.387+0000 7f24f4825640 1 -- start start 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.387+0000 7f24f4825640 1 librados: setting wanted keys 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.387+0000 7f24f4825640 1 librados: calling monclient init 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.387+0000 7f24f4825640 10 monclient: init 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.388+0000 7f24f4825640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.388+0000 7f24f4825640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.388+0000 7f24f4825640 10 monclient(hunting): picked mon.noname-b con 0x7f24f00aabc0 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.388+0000 7f24f4825640 10 monclient(hunting): picked mon.noname-c con 0x7f24f00ad0b0 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.388+0000 7f24f4825640 10 monclient(hunting): picked mon.noname-a con 0x7f24f00ad530 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.388+0000 7f24f4825640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f013d960 con 0x7f24f00ad530 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.388+0000 7f24f4825640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f013e9c0 con 0x7f24f00aabc0 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.388+0000 7f24f4825640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f0071a00 con 0x7f24f00ad0b0 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.388+0000 7f24f4825640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:28.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.388+0000 7f24f4825640 10 monclient(hunting): authenticate will time out at 960.585396s 2024-03-29T11:12:28.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.392+0000 7f24edd9b640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f24f00ad0b0 0x7f24f006fb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:50453/0 (socket says 172.21.15.157:50453) 2024-03-29T11:12:28.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.392+0000 7f24edd9b640 1 -- 172.21.15.157:0/457516167 learned_addr learned my addr 172.21.15.157:0/457516167 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:28.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.392+0000 7f24deffd640 10 client.?.objecter ms_handle_connect 0x7f24f00ad0b0 2024-03-29T11:12:28.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.393+0000 7f24deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 client.?.objecter ms_handle_connect 0x7f24f00ad530 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 client.?.objecter ms_handle_connect 0x7f24f00aabc0 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145029709 0 0) 0x7f24f013d960 con 0x7f24f00ad530 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d0004eb0 con 0x7f24f00ad530 2024-03-29T11:12:28.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991758825 0 0) 0x7f24f013e9c0 con 0x7f24f00aabc0 2024-03-29T11:12:28.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:28.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:28.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:28.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:12:28.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.395+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f013d960 con 0x7f24f00aabc0 2024-03-29T11:12:28.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.396+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971476017 0 0) 0x7f24f0071a00 con 0x7f24f00ad0b0 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.396+0000 7f24deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.396+0000 7f24deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.396+0000 7f24deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.396+0000 7f24deffd640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.396+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f013e9c0 con 0x7f24f00ad0b0 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1885013941 0 0) 0x7f24d0004eb0 con 0x7f24f00ad530 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d00066d0 con 0x7f24f00ad530 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1331775 0 0) 0x7f24f013d960 con 0x7f24f00aabc0 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d0004eb0 con 0x7f24f00aabc0 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f24e40017a0 con 0x7f24f00aabc0 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 10 monclient(hunting): dump: 2024-03-29T11:12:28.376 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f24e0001ae0 con 0x7f24f00ad530 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:28.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 10 monclient(hunting): dump: 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4161572124 0 0) 0x7f24f013e9c0 con 0x7f24f00ad0b0 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24f013d960 con 0x7f24f00ad0b0 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f24e8001d80 con 0x7f24f00ad0b0 2024-03-29T11:12:28.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.397+0000 7f24deffd640 10 monclient(hunting): dump: 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:28.379 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:28.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.398+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 531177455 0 0) 0x7f24d0004eb0 con 0x7f24f00aabc0 2024-03-29T11:12:28.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.398+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 >> v1:172.21.15.157:6790/0 conn(0x7f24f00ad0b0 legacy=0x7f24f006fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.398+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 >> v1:172.21.15.157:6789/0 conn(0x7f24f00ad530 legacy=0x7f24f01f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24deffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:28.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24deffd640 1 monclient: found mon.b 2024-03-29T11:12:28.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:28.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f0071b90 con 0x7f24f00aabc0 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24deffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24deffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24f4825640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24f4825640 10 monclient: _renew_subs 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24f4825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24f4825640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f013d480 con 0x7f24f00aabc0 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24f4825640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24f4825640 10 monclient: _renew_subs 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24f4825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24f4825640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f00727f0 con 0x7f24f00aabc0 2024-03-29T11:12:28.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.399+0000 7f24f4825640 1 librados: init done 2024-03-29T11:12:28.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.400+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24e4001bb0 con 0x7f24f00aabc0 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.400+0000 7f24deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.400+0000 7f24ecd99640 4 set_mon_vals no callback set 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.401+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f24e4004980 con 0x7f24f00aabc0 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.401+0000 7f24deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.401+0000 7f24f4825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.401+0000 7f24deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.401+0000 7f24deffd640 10 monclient: dump: 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:28.384 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.402+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f24e4004ae0 con 0x7f24f00aabc0 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.403+0000 7f24f4825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.403+0000 7f24f4825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.403+0000 7f24f4825640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d4005080 con 0x7f24f00aabc0 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.403+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f24e4084880 con 0x7f24f00aabc0 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.403+0000 7f24deffd640 10 client.4267.objecter ms_dispatch 0x7f24f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.403+0000 7f24deffd640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.403+0000 7f24deffd640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:28.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.403+0000 7f24deffd640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-29T11:12:28.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.406+0000 7f24edd9b640 10 monclient: get_auth_request con 0x7f24d006e290 auth_method 0 2024-03-29T11:12:28.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.407+0000 7f24deffd640 10 client.4267.objecter ms_handle_connect 0x7f24d006e290 2024-03-29T11:12:28.392 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.414+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24e4050160 con 0x7f24f00aabc0 2024-03-29T11:12:28.392 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.414+0000 7f24deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:28.392 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.414+0000 7f24deffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:28.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.741+0000 7f24f4825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:28.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.741+0000 7f24f4825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:28.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.741+0000 7f24f4825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:28.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.741+0000 7f24f4825640 1 -- 172.21.15.157:0/457516167 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24d4005330 con 0x7f24f00aabc0 2024-03-29T11:12:28.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.743+0000 7f24deffd640 1 -- 172.21.15.157:0/457516167 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f24e4050160 con 0x7f24f00aabc0 2024-03-29T11:12:28.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.743+0000 7f24deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:28.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.743+0000 7f24deffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:28.722 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:12:28.722 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:28.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.747+0000 7f24f4825640 10 librados: watch_flush enter 2024-03-29T11:12:28.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.748+0000 7f24f4825640 10 librados: watch_flush exit 2024-03-29T11:12:28.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.748+0000 7f24f4825640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-29T11:12:28.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.748+0000 7f24f4825640 10 client.4267.objecter successfully canceled tick 2024-03-29T11:12:28.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.748+0000 7f24f4825640 1 -- 172.21.15.157:0/457516167 >> v1:172.21.15.157:6816/1747058247 conn(0x7f24d006e290 legacy=0x7f24d0070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.748+0000 7f24f4825640 10 monclient: shutdown 2024-03-29T11:12:28.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.748+0000 7f24f4825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:28.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.748+0000 7f24f4825640 1 -- 172.21.15.157:0/457516167 >> v1:172.21.15.195:6789/0 conn(0x7f24f00aabc0 legacy=0x7f24f006f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:28.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.752+0000 7f24f4825640 1 -- 172.21.15.157:0/457516167 shutdown_connections 2024-03-29T11:12:28.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.752+0000 7f24f4825640 1 -- 172.21.15.157:0/457516167 >> 172.21.15.157:0/457516167 conn(0x7f24f00b3c80 msgr2=0x7f24f00b0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:28.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.752+0000 7f24f4825640 1 -- 172.21.15.157:0/457516167 shutdown_connections 2024-03-29T11:12:28.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.752+0000 7f24f4825640 1 -- 172.21.15.157:0/457516167 wait complete. 2024-03-29T11:12:28.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:28.752+0000 7f24f4825640 1 librados: shutdown 2024-03-29T11:12:28.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:35.556 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:35.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.733+0000 7ff8494c5640 10 monclient: get_monmap_and_config 2024-03-29T11:12:35.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.733+0000 7ff8494c5640 10 monclient: build_initial_monmap 2024-03-29T11:12:35.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.733+0000 7ff8494c5640 10 monclient: monmap: 2024-03-29T11:12:35.713 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:35.713 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:35.713 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:12:35.734688+0000 2024-03-29T11:12:35.713 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:12:35.734688+0000 2024-03-29T11:12:35.713 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:35.713 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:35.714 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:35.714 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:12:35.714 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:12:35.714 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:35.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.735+0000 7ff8494c5640 1 Processor -- start 2024-03-29T11:12:35.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.735+0000 7ff8494c5640 1 -- start start 2024-03-29T11:12:35.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.735+0000 7ff8494c5640 10 monclient: init 2024-03-29T11:12:35.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.736+0000 7ff8494c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:35.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.736+0000 7ff8494c5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:35.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.736+0000 7ff8494c5640 10 monclient(hunting): picked mon.noname-c con 0x7ff8440b0770 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:35.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.737+0000 7ff8494c5640 10 monclient(hunting): picked mon.noname-b con 0x7ff84414a700 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:35.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.737+0000 7ff8494c5640 10 monclient(hunting): picked mon.noname-a con 0x7ff8440ac5a0 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:35.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.737+0000 7ff8494c5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8440aba70 con 0x7ff8440ac5a0 2024-03-29T11:12:35.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.737+0000 7ff8494c5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8440aecb0 con 0x7ff8440b0770 2024-03-29T11:12:35.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.737+0000 7ff8494c5640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8440af150 con 0x7ff84414a700 2024-03-29T11:12:35.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.737+0000 7ff8494c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:35.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.737+0000 7ff8494c5640 10 monclient(hunting): authenticate will time out at 967.934545s 2024-03-29T11:12:35.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.739+0000 7ff8427fc640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ff84414a700 0x7ff8440abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:41231/0 (socket says 172.21.15.157:41231) 2024-03-29T11:12:35.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.740+0000 7ff8427fc640 1 -- 172.21.15.157:0/2501796067 learned_addr learned my addr 172.21.15.157:0/2501796067 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:35.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.740+0000 7ff841ffb640 1 -- 172.21.15.157:0/2501796067 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453013582 0 0) 0x7ff8440af150 con 0x7ff84414a700 2024-03-29T11:12:35.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.740+0000 7ff841ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:35.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.740+0000 7ff841ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:35.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.740+0000 7ff841ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:35.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.740+0000 7ff841ffb640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:12:35.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.740+0000 7ff841ffb640 1 -- 172.21.15.157:0/2501796067 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff82c003520 con 0x7ff84414a700 2024-03-29T11:12:35.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.740+0000 7ff841ffb640 1 -- 172.21.15.157:0/2501796067 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1065457318 0 0) 0x7ff82c003520 con 0x7ff84414a700 2024-03-29T11:12:35.719 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.740+0000 7ff841ffb640 1 -- 172.21.15.157:0/2501796067 >> v1:172.21.15.195:6789/0 conn(0x7ff8440b0770 legacy=0x7ff844149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:35.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff841ffb640 1 -- 172.21.15.157:0/2501796067 >> v1:172.21.15.157:6789/0 conn(0x7ff8440ac5a0 legacy=0x7ff844151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff841ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff841ffb640 1 monclient: found mon.noname-b 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff841ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff841ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff841ffb640 1 -- 172.21.15.157:0/2501796067 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8440ac9d0 con 0x7ff84414a700 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff841ffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff841ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff841ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff8494c5640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.741+0000 7ff8494c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.742+0000 7ff841ffb640 1 -- 172.21.15.157:0/2501796067 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff838001690 con 0x7ff84414a700 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.742+0000 7ff841ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.742+0000 7ff841ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:35.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.742+0000 7ff841ffb640 10 monclient: dump: 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.742+0000 7ff8494c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.742+0000 7ff841ffb640 1 -- 172.21.15.157:0/2501796067 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff838001920 con 0x7ff84414a700 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.742+0000 7ff841ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:35.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.742+0000 7ff8494c5640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:35.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.742+0000 7ff8494c5640 4 set_mon_vals no callback set 2024-03-29T11:12:35.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.743+0000 7ff8494c5640 10 monclient: shutdown 2024-03-29T11:12:35.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.743+0000 7ff8494c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:35.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.743+0000 7ff8494c5640 1 -- 172.21.15.157:0/2501796067 >> v1:172.21.15.157:6790/0 conn(0x7ff84414a700 legacy=0x7ff8440abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.743+0000 7ff8494c5640 1 -- 172.21.15.157:0/2501796067 shutdown_connections 2024-03-29T11:12:35.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.743+0000 7ff8494c5640 1 -- 172.21.15.157:0/2501796067 >> 172.21.15.157:0/2501796067 conn(0x7ff84406b660 msgr2=0x7ff84406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:35.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.748+0000 7ff8494c5640 1 -- 172.21.15.157:0/2501796067 shutdown_connections 2024-03-29T11:12:35.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.748+0000 7ff8494c5640 1 -- 172.21.15.157:0/2501796067 wait complete. 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.749+0000 7ff8494c5640 10 monclient: build_initial_monmap 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.749+0000 7ff8494c5640 10 monclient: monmap: 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:12:35.728 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:12:35.729 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:35.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.749+0000 7ff8494c5640 1 librados: starting msgr at 2024-03-29T11:12:35.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.749+0000 7ff8494c5640 1 librados: starting objecter 2024-03-29T11:12:35.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.750+0000 7ff8494c5640 1 Processor -- start 2024-03-29T11:12:35.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.750+0000 7ff8494c5640 1 -- start start 2024-03-29T11:12:35.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.750+0000 7ff8494c5640 1 librados: setting wanted keys 2024-03-29T11:12:35.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.750+0000 7ff8494c5640 1 librados: calling monclient init 2024-03-29T11:12:35.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.750+0000 7ff8494c5640 10 monclient: init 2024-03-29T11:12:35.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.751+0000 7ff8494c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:35.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.751+0000 7ff8494c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:35.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.751+0000 7ff8494c5640 10 monclient(hunting): picked mon.noname-b con 0x7ff8440ac5a0 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.754+0000 7ff8494c5640 10 monclient(hunting): picked mon.noname-a con 0x7ff8440b0770 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.754+0000 7ff8494c5640 10 monclient(hunting): picked mon.noname-c con 0x7ff84414a700 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.754+0000 7ff8494c5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8441db7f0 con 0x7ff8440b0770 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.754+0000 7ff8494c5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8441dc420 con 0x7ff8440ac5a0 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.754+0000 7ff8494c5640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8441dc5b0 con 0x7ff84414a700 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.754+0000 7ff8494c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.754+0000 7ff8494c5640 10 monclient(hunting): authenticate will time out at 967.951342s 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.755+0000 7ff8427fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff8440b0770 0x7ff844129060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:48375/0 (socket says 172.21.15.157:48375) 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.755+0000 7ff8427fc640 1 -- 172.21.15.157:0/3636392967 learned_addr learned my addr 172.21.15.157:0/3636392967 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.755+0000 7ff82b7fe640 10 client.?.objecter ms_handle_connect 0x7ff8440b0770 2024-03-29T11:12:35.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.755+0000 7ff82b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:35.736 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 10 client.?.objecter ms_handle_connect 0x7ff8440ac5a0 2024-03-29T11:12:35.736 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:35.736 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380640031 0 0) 0x7ff8441db7f0 con 0x7ff8440b0770 2024-03-29T11:12:35.736 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:35.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:35.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:35.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:12:35.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff830004eb0 con 0x7ff8440b0770 2024-03-29T11:12:35.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 262684823 0 0) 0x7ff830004eb0 con 0x7ff8440b0770 2024-03-29T11:12:35.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8441db7f0 con 0x7ff8440b0770 2024-03-29T11:12:35.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485039569 0 0) 0x7ff8441dc420 con 0x7ff8440ac5a0 2024-03-29T11:12:35.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.757+0000 7ff82b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff830004eb0 con 0x7ff8440ac5a0 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff838004a00 con 0x7ff8440b0770 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient(hunting): dump: 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:35.738 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1399526079 0 0) 0x7ff8441db7f0 con 0x7ff8440b0770 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 >> v1:172.21.15.157:6790/0 conn(0x7ff84414a700 legacy=0x7ff8441dbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 >> v1:172.21.15.195:6789/0 conn(0x7ff8440ac5a0 legacy=0x7ff8441282b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 1 monclient: found mon.a 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:35.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:12:35.740 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8441257e0 con 0x7ff8440b0770 2024-03-29T11:12:35.740 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:35.740 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:35.740 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:35.741 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff8494c5640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:12:35.741 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff8494c5640 10 monclient: _renew_subs 2024-03-29T11:12:35.741 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff8494c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:12:35.741 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff8494c5640 1 -- 172.21.15.157:0/3636392967 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff844125a30 con 0x7ff8440b0770 2024-03-29T11:12:35.741 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff838004f40 con 0x7ff8440b0770 2024-03-29T11:12:35.741 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:35.741 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff8494c5640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:35.741 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff8417fa640 4 set_mon_vals no callback set 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff838005310 con 0x7ff8440b0770 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff82b7fe640 10 monclient: dump: 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:35.742 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff8494c5640 10 monclient: _renew_subs 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff8494c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.758+0000 7ff8494c5640 1 -- 172.21.15.157:0/3636392967 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff844147420 con 0x7ff8440b0770 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.759+0000 7ff8494c5640 1 librados: init done 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.760+0000 7ff8494c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.760+0000 7ff8494c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.760+0000 7ff8494c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.760+0000 7ff8494c5640 1 -- 172.21.15.157:0/3636392967 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8440ab560 con 0x7ff8440b0770 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.761+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7ff838001b10 con 0x7ff8440b0770 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.762+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7ff838084d70 con 0x7ff8440b0770 2024-03-29T11:12:35.743 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.762+0000 7ff82b7fe640 10 client.4272.objecter ms_dispatch 0x7ff844000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:35.744 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.762+0000 7ff82b7fe640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:35.744 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.762+0000 7ff82b7fe640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:35.744 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.762+0000 7ff82b7fe640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-29T11:12:35.744 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.762+0000 7ff842ffd640 10 monclient: get_auth_request con 0x7ff83006dc80 auth_method 0 2024-03-29T11:12:35.744 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.763+0000 7ff82b7fe640 10 client.4272.objecter ms_handle_connect 0x7ff83006dc80 2024-03-29T11:12:35.752 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.772+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff838050650 con 0x7ff8440b0770 2024-03-29T11:12:35.752 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.772+0000 7ff82b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:35.752 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:35.772+0000 7ff82b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:36.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.099+0000 7ff8494c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:36.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.099+0000 7ff8494c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:36.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.099+0000 7ff8494c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:12:36.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.099+0000 7ff8494c5640 1 -- 172.21.15.157:0/3636392967 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8441db980 con 0x7ff8440b0770 2024-03-29T11:12:36.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.100+0000 7ff82b7fe640 1 -- 172.21.15.157:0/3636392967 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7ff838052580 con 0x7ff8440b0770 2024-03-29T11:12:36.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.100+0000 7ff82b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:36.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.100+0000 7ff82b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:36.079 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:12:36.079 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.104+0000 7ff8494c5640 10 librados: watch_flush enter 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.104+0000 7ff8494c5640 10 librados: watch_flush exit 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.104+0000 7ff8494c5640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.104+0000 7ff8494c5640 10 client.4272.objecter successfully canceled tick 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.104+0000 7ff8494c5640 1 -- 172.21.15.157:0/3636392967 >> v1:172.21.15.157:6816/1747058247 conn(0x7ff83006dc80 legacy=0x7ff830070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.104+0000 7ff8494c5640 10 monclient: shutdown 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.104+0000 7ff8494c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.104+0000 7ff8494c5640 1 -- 172.21.15.157:0/3636392967 >> v1:172.21.15.157:6789/0 conn(0x7ff8440b0770 legacy=0x7ff844129060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.104+0000 7ff8494c5640 1 -- 172.21.15.157:0/3636392967 shutdown_connections 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.104+0000 7ff8494c5640 1 -- 172.21.15.157:0/3636392967 >> 172.21.15.157:0/3636392967 conn(0x7ff84406b660 msgr2=0x7ff8440b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:36.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.105+0000 7ff8494c5640 1 -- 172.21.15.157:0/3636392967 shutdown_connections 2024-03-29T11:12:36.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.105+0000 7ff8494c5640 1 -- 172.21.15.157:0/3636392967 wait complete. 2024-03-29T11:12:36.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:36.105+0000 7ff8494c5640 1 librados: shutdown 2024-03-29T11:12:36.102 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:42.910 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:43.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.098+0000 7f242ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.098+0000 7f242ffff640 10 monclient: build_initial_monmap 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.098+0000 7f242ffff640 10 monclient: monmap: 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:12:43.099258+0000 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:12:43.099258+0000 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:12:43.077 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:43.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.102+0000 7f242ffff640 1 Processor -- start 2024-03-29T11:12:43.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.102+0000 7f242ffff640 1 -- start start 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.102+0000 7f242ffff640 10 monclient: init 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.103+0000 7f242ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.103+0000 7f242ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.103+0000 7f242ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f2428034850 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.103+0000 7f242ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f24280c9550 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.103+0000 7f242ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f242802aac0 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.103+0000 7f242ffff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242802c440 con 0x7f24280c9550 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.103+0000 7f242ffff640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242802aed0 con 0x7f2428034850 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.103+0000 7f242ffff640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242802b190 con 0x7f242802aac0 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.103+0000 7f242ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:43.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.103+0000 7f242ffff640 10 monclient(hunting): authenticate will time out at 975.300102s 2024-03-29T11:12:43.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.108+0000 7f242e7fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f24280c9550 0x7f24280cbea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:49149/0 (socket says 172.21.15.157:49149) 2024-03-29T11:12:43.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.108+0000 7f242e7fc640 1 -- 172.21.15.157:0/4129695814 learned_addr learned my addr 172.21.15.157:0/4129695814 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:43.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.110+0000 7f242dffb640 1 -- 172.21.15.157:0/4129695814 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423103685 0 0) 0x7f242802c440 con 0x7f24280c9550 2024-03-29T11:12:43.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.110+0000 7f242dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.110+0000 7f242dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.110+0000 7f242dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.110+0000 7f242dffb640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:12:43.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.110+0000 7f242dffb640 1 -- 172.21.15.157:0/4129695814 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2418003520 con 0x7f24280c9550 2024-03-29T11:12:43.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.110+0000 7f242dffb640 1 -- 172.21.15.157:0/4129695814 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196788308 0 0) 0x7f2418003520 con 0x7f24280c9550 2024-03-29T11:12:43.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 1 -- 172.21.15.157:0/4129695814 >> v1:172.21.15.157:6790/0 conn(0x7f242802aac0 legacy=0x7f24280cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 1 -- 172.21.15.157:0/4129695814 >> v1:172.21.15.195:6789/0 conn(0x7f2428034850 legacy=0x7f24280c8e40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 1 monclient: found mon.noname-a 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 1 -- 172.21.15.157:0/4129695814 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2428033060 con 0x7f24280c9550 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242ffff640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 1 -- 172.21.15.157:0/4129695814 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f243004f730 con 0x7f24280c9550 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:43.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 10 monclient: dump: 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 1 -- 172.21.15.157:0/4129695814 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24300008d0 con 0x7f24280c9550 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:43.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.111+0000 7f242ffff640 4 set_mon_vals no callback set 2024-03-29T11:12:43.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.112+0000 7f242dffb640 1 -- 172.21.15.157:0/4129695814 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f24300758b0 con 0x7f24280c9550 2024-03-29T11:12:43.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.112+0000 7f242dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.112+0000 7f242dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:43.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.112+0000 7f242dffb640 10 monclient: dump: 2024-03-29T11:12:43.093 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:43.093 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:43.093 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:43.093 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:43.093 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.094 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:43.094 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:43.094 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:43.094 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:43.094 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:43.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.112+0000 7f242ffff640 10 monclient: shutdown 2024-03-29T11:12:43.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.112+0000 7f242ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:43.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.112+0000 7f242ffff640 1 -- 172.21.15.157:0/4129695814 >> v1:172.21.15.157:6789/0 conn(0x7f24280c9550 legacy=0x7f24280cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.113+0000 7f242ffff640 1 -- 172.21.15.157:0/4129695814 shutdown_connections 2024-03-29T11:12:43.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.113+0000 7f242ffff640 1 -- 172.21.15.157:0/4129695814 >> 172.21.15.157:0/4129695814 conn(0x7f24280058f0 msgr2=0x7f2428005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:43.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.115+0000 7f242ffff640 1 -- 172.21.15.157:0/4129695814 shutdown_connections 2024-03-29T11:12:43.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.115+0000 7f242ffff640 1 -- 172.21.15.157:0/4129695814 wait complete. 2024-03-29T11:12:43.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.116+0000 7f242ffff640 10 monclient: build_initial_monmap 2024-03-29T11:12:43.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.116+0000 7f242ffff640 10 monclient: monmap: 2024-03-29T11:12:43.097 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:43.097 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:43.097 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:12:43.097 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:12:43.097 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:43.098 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:43.098 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:43.098 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:12:43.098 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:12:43.098 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:43.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.116+0000 7f242ffff640 1 librados: starting msgr at 2024-03-29T11:12:43.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.116+0000 7f242ffff640 1 librados: starting objecter 2024-03-29T11:12:43.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.117+0000 7f242ffff640 1 Processor -- start 2024-03-29T11:12:43.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.117+0000 7f242ffff640 1 -- start start 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.117+0000 7f242ffff640 1 librados: setting wanted keys 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.117+0000 7f242ffff640 1 librados: calling monclient init 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.117+0000 7f242ffff640 10 monclient: init 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.118+0000 7f242ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.118+0000 7f242ffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.118+0000 7f242ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f242802aac0 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.118+0000 7f242ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f2428034850 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.118+0000 7f242ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f24280c5590 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.118+0000 7f242ffff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242802f030 con 0x7f2428034850 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.118+0000 7f242ffff640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242802f2a0 con 0x7f24280c5590 2024-03-29T11:12:43.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.118+0000 7f242ffff640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242802dfb0 con 0x7f242802aac0 2024-03-29T11:12:43.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.118+0000 7f242ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:43.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.118+0000 7f242ffff640 10 monclient(hunting): authenticate will time out at 975.315069s 2024-03-29T11:12:43.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.121+0000 7f242effd640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f242802aac0 0x7f242802d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:51703/0 (socket says 172.21.15.157:51703) 2024-03-29T11:12:43.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.121+0000 7f242effd640 1 -- 172.21.15.157:0/2228414031 learned_addr learned my addr 172.21.15.157:0/2228414031 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:43.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.121+0000 7f24177fe640 10 client.?.objecter ms_handle_connect 0x7f242802aac0 2024-03-29T11:12:43.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.121+0000 7f24177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:43.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.121+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996223950 0 0) 0x7f242802dfb0 con 0x7f242802aac0 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.121+0000 7f24177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.121+0000 7f24177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.121+0000 7f24177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.121+0000 7f24177fe640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.121+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f241c004eb0 con 0x7f242802aac0 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.122+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2164722496 0 0) 0x7f241c004eb0 con 0x7f242802aac0 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.122+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f241c005300 con 0x7f242802aac0 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.122+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2424001790 con 0x7f242802aac0 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.122+0000 7f24177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.122+0000 7f24177fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.122+0000 7f24177fe640 10 monclient(hunting): dump: 2024-03-29T11:12:43.103 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.122+0000 7f24177fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.122+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3280839898 0 0) 0x7f241c005300 con 0x7f242802aac0 2024-03-29T11:12:43.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.123+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 >> v1:172.21.15.195:6789/0 conn(0x7f24280c5590 legacy=0x7f2428160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.123+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 >> v1:172.21.15.157:6789/0 conn(0x7f2428034850 legacy=0x7f2428031cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:43.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:43.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 1 monclient: found mon.c 2024-03-29T11:12:43.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:43.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:43.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242802e1d0 con 0x7f242802aac0 2024-03-29T11:12:43.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:43.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 10 client.?.objecter ms_handle_connect 0x7f2428034850 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2424001d60 con 0x7f242802aac0 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2424001790 con 0x7f242802aac0 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f24177fe640 10 monclient: dump: 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:43.106 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:43.107 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:43.107 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:43.107 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.107 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:43.107 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:43.107 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:43.107 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:43.107 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:43.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.125+0000 7f242d7fa640 4 set_mon_vals no callback set 2024-03-29T11:12:43.109 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.128+0000 7f242ffff640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:12:43.109 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.128+0000 7f242ffff640 10 monclient: _renew_subs 2024-03-29T11:12:43.109 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.128+0000 7f242ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:43.109 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.128+0000 7f242ffff640 1 -- 172.21.15.157:0/2228414031 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2428032400 con 0x7f242802aac0 2024-03-29T11:12:43.109 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.128+0000 7f242ffff640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:43.109 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.128+0000 7f242ffff640 10 monclient: _renew_subs 2024-03-29T11:12:43.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.128+0000 7f242ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:43.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.128+0000 7f242ffff640 1 -- 172.21.15.157:0/2228414031 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24281615d0 con 0x7f242802aac0 2024-03-29T11:12:43.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.128+0000 7f242ffff640 1 librados: init done 2024-03-29T11:12:43.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.130+0000 7f24157fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:43.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.130+0000 7f24157fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:43.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.130+0000 7f24157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:43.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.130+0000 7f24157fa640 1 -- 172.21.15.157:0/2228414031 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242802dd30 con 0x7f242802aac0 2024-03-29T11:12:43.111 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.131+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f242401a810 con 0x7f242802aac0 2024-03-29T11:12:43.111 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.132+0000 7f242e7fc640 10 monclient: get_auth_request con 0x7f241c06e1d0 auth_method 0 2024-03-29T11:12:43.111 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.133+0000 7f24177fe640 10 client.4280.objecter ms_handle_connect 0x7f241c06e1d0 2024-03-29T11:12:43.112 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.133+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f24240543d0 con 0x7f242802aac0 2024-03-29T11:12:43.112 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.133+0000 7f24177fe640 10 client.4280.objecter ms_dispatch 0x7f2430000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:43.112 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.133+0000 7f24177fe640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:43.112 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.133+0000 7f24177fe640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:43.112 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.133+0000 7f24177fe640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T11:12:43.124 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.146+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2424050670 con 0x7f242802aac0 2024-03-29T11:12:43.125 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.146+0000 7f24177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:43.125 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.146+0000 7f24177fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:43.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.470+0000 7f24157fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:43.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.470+0000 7f24157fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:43.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.470+0000 7f24157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:43.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.470+0000 7f24157fa640 1 -- 172.21.15.157:0/2228414031 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2428004cd0 con 0x7f242802aac0 2024-03-29T11:12:43.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.474+0000 7f24177fe640 1 -- 172.21.15.157:0/2228414031 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f2424050670 con 0x7f242802aac0 2024-03-29T11:12:43.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.474+0000 7f24177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:43.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.474+0000 7f24177fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:43.453 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:12:43.453 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:43.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.477+0000 7f24157fa640 10 librados: watch_flush enter 2024-03-29T11:12:43.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.478+0000 7f24157fa640 10 librados: watch_flush exit 2024-03-29T11:12:43.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.478+0000 7f24157fa640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T11:12:43.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.478+0000 7f24157fa640 10 client.4280.objecter successfully canceled tick 2024-03-29T11:12:43.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.478+0000 7f24157fa640 1 -- 172.21.15.157:0/2228414031 >> v1:172.21.15.157:6816/1747058247 conn(0x7f241c06e1d0 legacy=0x7f241c070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.478+0000 7f24157fa640 10 monclient: shutdown 2024-03-29T11:12:43.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.478+0000 7f24157fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:43.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.478+0000 7f24157fa640 1 -- 172.21.15.157:0/2228414031 >> v1:172.21.15.157:6790/0 conn(0x7f242802aac0 legacy=0x7f242802d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.478+0000 7f24157fa640 1 -- 172.21.15.157:0/2228414031 shutdown_connections 2024-03-29T11:12:43.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.478+0000 7f24157fa640 1 -- 172.21.15.157:0/2228414031 >> 172.21.15.157:0/2228414031 conn(0x7f24280058f0 msgr2=0x7f24280043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:43.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.479+0000 7f24157fa640 1 -- 172.21.15.157:0/2228414031 shutdown_connections 2024-03-29T11:12:43.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.479+0000 7f24157fa640 1 -- 172.21.15.157:0/2228414031 wait complete. 2024-03-29T11:12:43.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:43.479+0000 7f24157fa640 1 librados: shutdown 2024-03-29T11:12:43.477 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:50.290 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:50.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.468+0000 7fdaaf997640 10 monclient: get_monmap_and_config 2024-03-29T11:12:50.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.468+0000 7fdaaf997640 10 monclient: build_initial_monmap 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.468+0000 7fdaaf997640 10 monclient: monmap: 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:12:50.469993+0000 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:12:50.469993+0000 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:12:50.447 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:50.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.469+0000 7fdaaf997640 1 Processor -- start 2024-03-29T11:12:50.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.470+0000 7fdaaf997640 1 -- start start 2024-03-29T11:12:50.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.470+0000 7fdaaf997640 10 monclient: init 2024-03-29T11:12:50.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaaf997640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:50.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaaf997640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:50.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaaf997640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa81457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:50.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaaf997640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa813f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:50.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaaf997640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa80a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:50.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaad733640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fdaa81457f0 0x7fdaa813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:52298/0 (socket says 172.21.15.157:52298) 2024-03-29T11:12:50.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaad733640 1 -- 172.21.15.157:0/1386080433 learned_addr learned my addr 172.21.15.157:0/1386080433 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:50.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaaf997640 1 -- 172.21.15.157:0/1386080433 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa80aaef0 con 0x7fdaa81457f0 2024-03-29T11:12:50.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaaf997640 1 -- 172.21.15.157:0/1386080433 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa8145bd0 con 0x7fdaa813f650 2024-03-29T11:12:50.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaaf997640 1 -- 172.21.15.157:0/1386080433 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa8145eb0 con 0x7fdaa80a9d70 2024-03-29T11:12:50.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaaf997640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:50.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.471+0000 7fdaaf997640 10 monclient(hunting): authenticate will time out at 982.668785s 2024-03-29T11:12:50.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038253575 0 0) 0x7fdaa80aaef0 con 0x7fdaa81457f0 2024-03-29T11:12:50.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:50.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda94003520 con 0x7fdaa81457f0 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346611503 0 0) 0x7fdaa8145bd0 con 0x7fdaa813f650 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa80aaef0 con 0x7fdaa813f650 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498759431 0 0) 0x7fdaa8145eb0 con 0x7fdaa80a9d70 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:50.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa8145bd0 con 0x7fdaa80a9d70 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 611114756 0 0) 0x7fda94003520 con 0x7fdaa81457f0 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 >> v1:172.21.15.157:6790/0 conn(0x7fdaa80a9d70 legacy=0x7fdaa814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 >> v1:172.21.15.195:6789/0 conn(0x7fdaa813f650 legacy=0x7fdaa80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 monclient: found mon.noname-a 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.472+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa8146340 con 0x7fdaa81457f0 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 10 monclient: _finish_auth 0 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fdaaf997640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdaa4001690 con 0x7fdaa81457f0 2024-03-29T11:12:50.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 10 monclient: dump: 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fdaaf997640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:50.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdaa40018b0 con 0x7fdaa81457f0 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fdaaf997640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 1 -- 172.21.15.157:0/1386080433 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdaa4001c40 con 0x7fdaa81457f0 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fda9ffff640 10 monclient: dump: 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:50.457 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.473+0000 7fdaaf997640 4 set_mon_vals no callback set 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.474+0000 7fdaaf997640 10 monclient: shutdown 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.474+0000 7fdaaf997640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.474+0000 7fdaaf997640 1 -- 172.21.15.157:0/1386080433 >> v1:172.21.15.157:6789/0 conn(0x7fdaa81457f0 legacy=0x7fdaa813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.474+0000 7fdaaf997640 1 -- 172.21.15.157:0/1386080433 shutdown_connections 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.474+0000 7fdaaf997640 1 -- 172.21.15.157:0/1386080433 >> 172.21.15.157:0/1386080433 conn(0x7fdaa8066a00 msgr2=0x7fdaa8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.474+0000 7fdaaf997640 1 -- 172.21.15.157:0/1386080433 shutdown_connections 2024-03-29T11:12:50.458 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.474+0000 7fdaaf997640 1 -- 172.21.15.157:0/1386080433 wait complete. 2024-03-29T11:12:50.462 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.475+0000 7fdaaf997640 10 monclient: build_initial_monmap 2024-03-29T11:12:50.462 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.475+0000 7fdaaf997640 10 monclient: monmap: 2024-03-29T11:12:50.463 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:50.463 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:50.463 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:12:50.463 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:12:50.463 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.475+0000 7fdaaf997640 1 librados: starting msgr at 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.475+0000 7fdaaf997640 1 librados: starting objecter 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.475+0000 7fdaaf997640 1 Processor -- start 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.475+0000 7fdaaf997640 1 -- start start 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.475+0000 7fdaaf997640 1 librados: setting wanted keys 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.475+0000 7fdaaf997640 1 librados: calling monclient init 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.475+0000 7fdaaf997640 10 monclient: init 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaaf997640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:50.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaaf997640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaaf997640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa80a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaaf997640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa813f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaaf997640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa81457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaaf997640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa81418c0 con 0x7fdaa80a9d70 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaaf997640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa8139160 con 0x7fdaa81457f0 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaaf997640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa8135a00 con 0x7fdaa813f650 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaaf997640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaaf997640 10 monclient(hunting): authenticate will time out at 982.673767s 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.476+0000 7fdaacf32640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fdaa813f650 0x7fdaa8134be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:47586/0 (socket says 172.21.15.157:47586) 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fdaacf32640 1 -- 172.21.15.157:0/1784808736 learned_addr learned my addr 172.21.15.157:0/1784808736 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 10 client.?.objecter ms_handle_connect 0x7fdaa813f650 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 10 client.?.objecter ms_handle_connect 0x7fdaa81457f0 2024-03-29T11:12:50.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018209278 0 0) 0x7fdaa8139160 con 0x7fdaa81457f0 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda8c004eb0 con 0x7fdaa81457f0 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 10 client.?.objecter ms_handle_connect 0x7fdaa80a9d70 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.477+0000 7fda9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719363352 0 0) 0x7fdaa8135a00 con 0x7fdaa813f650 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:50.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa8139160 con 0x7fdaa813f650 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163773666 0 0) 0x7fdaa81418c0 con 0x7fdaa80a9d70 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa8135a00 con 0x7fdaa80a9d70 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2218243502 0 0) 0x7fda8c004eb0 con 0x7fdaa81457f0 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaa81418c0 con 0x7fdaa81457f0 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdaa0001bf0 con 0x7fdaa81457f0 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): dump: 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:50.467 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.478+0000 7fda9dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3860847598 0 0) 0x7fdaa81418c0 con 0x7fdaa81457f0 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 >> v1:172.21.15.157:6790/0 conn(0x7fdaa813f650 legacy=0x7fdaa8134be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 >> v1:172.21.15.157:6789/0 conn(0x7fdaa80a9d70 legacy=0x7fdaa81402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 1 monclient: found mon.b 2024-03-29T11:12:50.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa8141b30 con 0x7fdaa81457f0 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fdaaf997640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fdaaf997640 10 monclient: _renew_subs 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fdaaf997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fdaaf997640 1 -- 172.21.15.157:0/1784808736 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa8141e10 con 0x7fdaa81457f0 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fdaaf997640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fdaaf997640 10 monclient: _renew_subs 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fdaaf997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fdaaf997640 1 -- 172.21.15.157:0/1784808736 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa813ba70 con 0x7fdaa81457f0 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fdaaf997640 1 librados: init done 2024-03-29T11:12:50.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdaa00020e0 con 0x7fdaa81457f0 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdaa0003b30 con 0x7fdaa81457f0 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9dffb640 10 monclient: dump: 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:50.470 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.479+0000 7fda9f7fe640 4 set_mon_vals no callback set 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.481+0000 7fdaaf997640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.481+0000 7fdaaf997640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.481+0000 7fdaaf997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.481+0000 7fdaaf997640 1 -- 172.21.15.157:0/1784808736 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda88005080 con 0x7fdaa81457f0 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.482+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fdaa0019b00 con 0x7fdaa81457f0 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.482+0000 7fdaad733640 10 monclient: get_auth_request con 0x7fda8c06dd80 auth_method 0 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.482+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fdaa0083bb0 con 0x7fdaa81457f0 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.482+0000 7fda9dffb640 10 client.4276.objecter ms_dispatch 0x7fdaa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.482+0000 7fda9dffb640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.482+0000 7fda9dffb640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.483+0000 7fda9dffb640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-29T11:12:50.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.483+0000 7fda9dffb640 10 client.4276.objecter ms_handle_connect 0x7fda8c06dd80 2024-03-29T11:12:50.472 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.489+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdaa004f2c0 con 0x7fdaa81457f0 2024-03-29T11:12:50.472 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.489+0000 7fda9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:50.472 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.489+0000 7fda9dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:50.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.849+0000 7fdaaf997640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:50.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.849+0000 7fdaaf997640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:50.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.849+0000 7fdaaf997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:12:50.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.849+0000 7fdaaf997640 1 -- 172.21.15.157:0/1784808736 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda88005330 con 0x7fdaa81457f0 2024-03-29T11:12:50.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.850+0000 7fda9dffb640 1 -- 172.21.15.157:0/1784808736 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fdaa00513c0 con 0x7fdaa81457f0 2024-03-29T11:12:50.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.850+0000 7fda9dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:50.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.850+0000 7fda9dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:50.829 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:12:50.830 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:50.832 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.854+0000 7fdaaf997640 10 librados: watch_flush enter 2024-03-29T11:12:50.832 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.854+0000 7fdaaf997640 10 librados: watch_flush exit 2024-03-29T11:12:50.832 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.854+0000 7fdaaf997640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-29T11:12:50.832 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.854+0000 7fdaaf997640 10 client.4276.objecter successfully canceled tick 2024-03-29T11:12:50.833 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.854+0000 7fdaaf997640 1 -- 172.21.15.157:0/1784808736 >> v1:172.21.15.157:6816/1747058247 conn(0x7fda8c06dd80 legacy=0x7fda8c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:50.833 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.854+0000 7fdaaf997640 10 monclient: shutdown 2024-03-29T11:12:50.833 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.854+0000 7fdaaf997640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:50.833 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.854+0000 7fdaaf997640 1 -- 172.21.15.157:0/1784808736 >> v1:172.21.15.195:6789/0 conn(0x7fdaa81457f0 legacy=0x7fdaa81352f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:50.834 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.855+0000 7fdaaf997640 1 -- 172.21.15.157:0/1784808736 shutdown_connections 2024-03-29T11:12:50.834 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.855+0000 7fdaaf997640 1 -- 172.21.15.157:0/1784808736 >> 172.21.15.157:0/1784808736 conn(0x7fdaa8066a00 msgr2=0x7fdaa80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:50.834 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.855+0000 7fdaaf997640 1 -- 172.21.15.157:0/1784808736 shutdown_connections 2024-03-29T11:12:50.834 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.855+0000 7fdaaf997640 1 -- 172.21.15.157:0/1784808736 wait complete. 2024-03-29T11:12:50.834 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:50.855+0000 7fdaaf997640 1 librados: shutdown 2024-03-29T11:12:50.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:57.663 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:57.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.812+0000 7fd3ba5e8640 10 monclient: get_monmap_and_config 2024-03-29T11:12:57.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.812+0000 7fd3ba5e8640 10 monclient: build_initial_monmap 2024-03-29T11:12:57.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.812+0000 7fd3ba5e8640 10 monclient: monmap: 2024-03-29T11:12:57.790 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:57.790 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:57.790 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:12:57.813349+0000 2024-03-29T11:12:57.790 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:12:57.813349+0000 2024-03-29T11:12:57.790 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:57.790 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:57.791 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:57.791 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:12:57.791 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:12:57.791 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:57.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.813+0000 7fd3ba5e8640 1 Processor -- start 2024-03-29T11:12:57.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.813+0000 7fd3ba5e8640 1 -- start start 2024-03-29T11:12:57.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.813+0000 7fd3ba5e8640 10 monclient: init 2024-03-29T11:12:57.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.813+0000 7fd3ba5e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:57.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.813+0000 7fd3ba5e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:57.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.814+0000 7fd3ba5e8640 10 monclient(hunting): picked mon.noname-a con 0x7fd3b41457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:57.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.814+0000 7fd3ba5e8640 10 monclient(hunting): picked mon.noname-b con 0x7fd3b413f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:57.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.814+0000 7fd3ba5e8640 10 monclient(hunting): picked mon.noname-c con 0x7fd3b40a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:57.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.814+0000 7fd3ba5e8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b40aaef0 con 0x7fd3b41457f0 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.814+0000 7fd3ba5e8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b4145bd0 con 0x7fd3b40a9d70 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.814+0000 7fd3ba5e8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b4145eb0 con 0x7fd3b413f650 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.814+0000 7fd3ba5e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.814+0000 7fd3ba5e8640 10 monclient(hunting): authenticate will time out at 990.011619s 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.814+0000 7fd3b37fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fd3b413f650 0x7fd3b40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:42512/0 (socket says 172.21.15.157:42512) 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.814+0000 7fd3b37fe640 1 -- 172.21.15.157:0/1900432844 learned_addr learned my addr 172.21.15.157:0/1900432844 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.815+0000 7fd3b2ffd640 1 -- 172.21.15.157:0/1900432844 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014670361 0 0) 0x7fd3b4145eb0 con 0x7fd3b413f650 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.815+0000 7fd3b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.815+0000 7fd3b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.815+0000 7fd3b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.815+0000 7fd3b2ffd640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.815+0000 7fd3b2ffd640 1 -- 172.21.15.157:0/1900432844 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd39c003520 con 0x7fd3b413f650 2024-03-29T11:12:57.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.815+0000 7fd3b2ffd640 1 -- 172.21.15.157:0/1900432844 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 247621770 0 0) 0x7fd39c003520 con 0x7fd3b413f650 2024-03-29T11:12:57.795 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.815+0000 7fd3b2ffd640 1 -- 172.21.15.157:0/1900432844 >> v1:172.21.15.195:6789/0 conn(0x7fd3b40a9d70 legacy=0x7fd3b414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:57.795 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.815+0000 7fd3b2ffd640 1 -- 172.21.15.157:0/1900432844 >> v1:172.21.15.157:6789/0 conn(0x7fd3b41457f0 legacy=0x7fd3b413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.817+0000 7fd3b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.817+0000 7fd3b2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.817+0000 7fd3b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.817+0000 7fd3b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.817+0000 7fd3b2ffd640 1 -- 172.21.15.157:0/1900432844 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b4146340 con 0x7fd3b413f650 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.817+0000 7fd3b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.817+0000 7fd3b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 1 -- 172.21.15.157:0/1900432844 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd3a8001690 con 0x7fd3b413f650 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 10 monclient: dump: 2024-03-29T11:12:57.796 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3ba5e8640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3ba5e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 1 -- 172.21.15.157:0/1900432844 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3a8001960 con 0x7fd3b413f650 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:57.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 1 -- 172.21.15.157:0/1900432844 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd3a8001cf0 con 0x7fd3b413f650 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3b2ffd640 10 monclient: dump: 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:57.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3ba5e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:57.799 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.818+0000 7fd3ba5e8640 4 set_mon_vals no callback set 2024-03-29T11:12:57.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.819+0000 7fd3ba5e8640 10 monclient: shutdown 2024-03-29T11:12:57.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.819+0000 7fd3ba5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:57.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.819+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/1900432844 >> v1:172.21.15.157:6790/0 conn(0x7fd3b413f650 legacy=0x7fd3b40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:57.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.819+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/1900432844 shutdown_connections 2024-03-29T11:12:57.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.819+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/1900432844 >> 172.21.15.157:0/1900432844 conn(0x7fd3b4066a00 msgr2=0x7fd3b4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:57.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.819+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/1900432844 shutdown_connections 2024-03-29T11:12:57.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.819+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/1900432844 wait complete. 2024-03-29T11:12:57.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.820+0000 7fd3ba5e8640 10 monclient: build_initial_monmap 2024-03-29T11:12:57.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.820+0000 7fd3ba5e8640 10 monclient: monmap: 2024-03-29T11:12:57.801 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:12:57.801 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.820+0000 7fd3ba5e8640 1 librados: starting msgr at 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.820+0000 7fd3ba5e8640 1 librados: starting objecter 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.821+0000 7fd3ba5e8640 1 Processor -- start 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.821+0000 7fd3ba5e8640 1 -- start start 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.821+0000 7fd3ba5e8640 1 librados: setting wanted keys 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.821+0000 7fd3ba5e8640 1 librados: calling monclient init 2024-03-29T11:12:57.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.822+0000 7fd3ba5e8640 10 monclient: init 2024-03-29T11:12:57.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.823+0000 7fd3ba5e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:57.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.823+0000 7fd3ba5e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:57.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.823+0000 7fd3ba5e8640 10 monclient(hunting): picked mon.noname-b con 0x7fd3b40a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:12:57.809 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.823+0000 7fd3ba5e8640 10 monclient(hunting): picked mon.noname-a con 0x7fd3b413f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:12:57.809 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3ba5e8640 10 monclient(hunting): picked mon.noname-c con 0x7fd3b41457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:12:57.809 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b8b85640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fd3b41457f0 0x7fd3b41e1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:43170/0 (socket says 172.21.15.157:43170) 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b8b85640 1 -- 172.21.15.157:0/4009376427 learned_addr learned my addr 172.21.15.157:0/4009376427 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b40ac280 con 0x7fd3b413f650 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b40ac4f0 con 0x7fd3b40a9d70 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b0ff9640 10 client.?.objecter ms_handle_connect 0x7fd3b41457f0 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b40ac7d0 con 0x7fd3b41457f0 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3ba5e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3ba5e8640 10 monclient(hunting): authenticate will time out at 990.021215s 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452961137 0 0) 0x7fd3b40ac7d0 con 0x7fd3b41457f0 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b0ff9640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:12:57.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3a0004eb0 con 0x7fd3b41457f0 2024-03-29T11:12:57.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.824+0000 7fd3b0ff9640 10 client.?.objecter ms_handle_connect 0x7fd3b40a9d70 2024-03-29T11:12:57.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:57.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1046457534 0 0) 0x7fd3a0004eb0 con 0x7fd3b41457f0 2024-03-29T11:12:57.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3b40ac7d0 con 0x7fd3b41457f0 2024-03-29T11:12:57.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd3ac001cf0 con 0x7fd3b41457f0 2024-03-29T11:12:57.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:57.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:57.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 10 monclient(hunting): dump: 2024-03-29T11:12:57.811 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:57.811 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466988327 0 0) 0x7fd3b40ac4f0 con 0x7fd3b40a9d70 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:57.812 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3a0004eb0 con 0x7fd3b40a9d70 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448684443 0 0) 0x7fd3a0004eb0 con 0x7fd3b40a9d70 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.825+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3b40ac4f0 con 0x7fd3b40a9d70 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd3a4001690 con 0x7fd3b40a9d70 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 10 monclient(hunting): dump: 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:57.813 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2745739035 0 0) 0x7fd3b40ac7d0 con 0x7fd3b41457f0 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 >> v1:172.21.15.195:6789/0 conn(0x7fd3b40a9d70 legacy=0x7fd3b4144880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 >> v1:172.21.15.157:6789/0 conn(0x7fd3b413f650 legacy=0x7fd3b41d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 1 monclient: found mon.c 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b41d77e0 con 0x7fd3b41457f0 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:57.814 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 10 client.?.objecter ms_handle_connect 0x7fd3b413f650 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3ba5e8640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3ba5e8640 10 monclient: _renew_subs 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3ba5e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b4137c10 con 0x7fd3b41457f0 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3ba5e8640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3ba5e8640 10 monclient: _renew_subs 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3ba5e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b413fda0 con 0x7fd3b41457f0 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.826+0000 7fd3ba5e8640 1 librados: init done 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.827+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3ac0020e0 con 0x7fd3b41457f0 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.827+0000 7fd3b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.827+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd3ac003de0 con 0x7fd3b41457f0 2024-03-29T11:12:57.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.827+0000 7fd3b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.827+0000 7fd3b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.827+0000 7fd3b0ff9640 10 monclient: dump: 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.827+0000 7fd3b27fc640 4 set_mon_vals no callback set 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.827+0000 7fd3ba5e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:57.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.828+0000 7fd3ba5e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:57.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.828+0000 7fd3ba5e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:57.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.828+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd394005080 con 0x7fd3b41457f0 2024-03-29T11:12:57.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.828+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fd3ac003f40 con 0x7fd3b41457f0 2024-03-29T11:12:57.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.829+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fd3ac084ea0 con 0x7fd3b41457f0 2024-03-29T11:12:57.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.829+0000 7fd3b0ff9640 10 client.4292.objecter ms_dispatch 0x7fd3b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:57.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.829+0000 7fd3b0ff9640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:57.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.829+0000 7fd3b0ff9640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:57.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.829+0000 7fd3b3fff640 10 monclient: get_auth_request con 0x7fd3a006de80 auth_method 0 2024-03-29T11:12:57.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.829+0000 7fd3b0ff9640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T11:12:57.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.829+0000 7fd3b0ff9640 10 client.4292.objecter ms_handle_connect 0x7fd3a006de80 2024-03-29T11:12:57.818 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.836+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3ac050780 con 0x7fd3b41457f0 2024-03-29T11:12:57.818 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.836+0000 7fd3b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:57.818 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:57.836+0000 7fd3b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:58.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.107+0000 7fd3ba5e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:58.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.107+0000 7fd3ba5e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:58.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.107+0000 7fd3ba5e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:12:58.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.107+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd394005330 con 0x7fd3b41457f0 2024-03-29T11:12:58.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.108+0000 7fd3b0ff9640 1 -- 172.21.15.157:0/4009376427 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fd3ac0526b0 con 0x7fd3b41457f0 2024-03-29T11:12:58.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.108+0000 7fd3b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:58.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.108+0000 7fd3b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:58.086 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:12:58.086 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:58.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.111+0000 7fd3ba5e8640 10 librados: watch_flush enter 2024-03-29T11:12:58.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.111+0000 7fd3ba5e8640 10 librados: watch_flush exit 2024-03-29T11:12:58.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.111+0000 7fd3ba5e8640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T11:12:58.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.111+0000 7fd3ba5e8640 10 client.4292.objecter successfully canceled tick 2024-03-29T11:12:58.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.112+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 >> v1:172.21.15.157:6816/1747058247 conn(0x7fd3a006de80 legacy=0x7fd3a0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:58.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.112+0000 7fd3ba5e8640 10 monclient: shutdown 2024-03-29T11:12:58.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.112+0000 7fd3ba5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:58.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.112+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 >> v1:172.21.15.157:6790/0 conn(0x7fd3b41457f0 legacy=0x7fd3b41e1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:58.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.112+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 shutdown_connections 2024-03-29T11:12:58.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.112+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 >> 172.21.15.157:0/4009376427 conn(0x7fd3b4066a00 msgr2=0x7fd3b413a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:58.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.112+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 shutdown_connections 2024-03-29T11:12:58.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.112+0000 7fd3ba5e8640 1 -- 172.21.15.157:0/4009376427 wait complete. 2024-03-29T11:12:58.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:12:58.112+0000 7fd3ba5e8640 1 librados: shutdown 2024-03-29T11:12:58.103 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:04.911 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:05.060 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.082+0000 7f1765d74640 10 monclient: get_monmap_and_config 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.082+0000 7f1765d74640 10 monclient: build_initial_monmap 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.082+0000 7f1765d74640 10 monclient: monmap: 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:13:05.083987+0000 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:13:05.083987+0000 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:13:05.061 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:05.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.085+0000 7f1765d74640 1 Processor -- start 2024-03-29T11:13:05.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.085+0000 7f1765d74640 1 -- start start 2024-03-29T11:13:05.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.086+0000 7f1765d74640 10 monclient: init 2024-03-29T11:13:05.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.086+0000 7f1765d74640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:05.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.086+0000 7f1765d74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:05.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.086+0000 7f1765d74640 10 monclient(hunting): picked mon.noname-a con 0x7f1758006e30 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:05.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.086+0000 7f1765d74640 10 monclient(hunting): picked mon.noname-b con 0x7f1758035e00 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:05.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.087+0000 7f1765d74640 10 monclient(hunting): picked mon.noname-c con 0x7f17580c9670 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:05.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.087+0000 7f1765d74640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1758035970 con 0x7f1758006e30 2024-03-29T11:13:05.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.087+0000 7f1765d74640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1758036280 con 0x7f17580c9670 2024-03-29T11:13:05.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.087+0000 7f1765d74640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1758035640 con 0x7f1758035e00 2024-03-29T11:13:05.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.087+0000 7f1765d74640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:05.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.087+0000 7f1765d74640 10 monclient(hunting): authenticate will time out at 997.284379s 2024-03-29T11:13:05.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.090+0000 7f1764d72640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1758006e30 0x7f17580cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:52163/0 (socket says 172.21.15.157:52163) 2024-03-29T11:13:05.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.090+0000 7f1764d72640 1 -- 172.21.15.157:0/2401009331 learned_addr learned my addr 172.21.15.157:0/2401009331 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:05.072 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032666502 0 0) 0x7f1758036280 con 0x7f17580c9670 2024-03-29T11:13:05.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:05.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:05.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:05.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:13:05.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1750003520 con 0x7f17580c9670 2024-03-29T11:13:05.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149204046 0 0) 0x7f1758035640 con 0x7f1758035e00 2024-03-29T11:13:05.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:05.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:05.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:05.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:13:05.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.094+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1758036280 con 0x7f1758035e00 2024-03-29T11:13:05.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490509018 0 0) 0x7f1758035970 con 0x7f1758006e30 2024-03-29T11:13:05.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:05.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:05.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:05.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:13:05.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1758035640 con 0x7f1758006e30 2024-03-29T11:13:05.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4239999859 0 0) 0x7f1750003520 con 0x7f17580c9670 2024-03-29T11:13:05.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 >> v1:172.21.15.157:6790/0 conn(0x7f1758035e00 legacy=0x7f17580c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:05.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 >> v1:172.21.15.157:6789/0 conn(0x7f1758006e30 legacy=0x7f17580cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:05.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:05.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:13:05.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:05.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:13:05.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1758034e20 con 0x7f17580c9670 2024-03-29T11:13:05.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:05.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:05.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:05.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1754001540 con 0x7f17580c9670 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.098+0000 7f175f7fe640 10 monclient: dump: 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:05.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.099+0000 7f1765d74640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:13:05.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.099+0000 7f1765d74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:05.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.101+0000 7f175f7fe640 1 -- 172.21.15.157:0/2401009331 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17540016a0 con 0x7f17580c9670 2024-03-29T11:13:05.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.101+0000 7f175f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:05.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.101+0000 7f1765d74640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:05.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.101+0000 7f1765d74640 4 set_mon_vals no callback set 2024-03-29T11:13:05.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.102+0000 7f1765d74640 10 monclient: shutdown 2024-03-29T11:13:05.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.102+0000 7f1765d74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:05.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.102+0000 7f1765d74640 1 -- 172.21.15.157:0/2401009331 >> v1:172.21.15.195:6789/0 conn(0x7f17580c9670 legacy=0x7f17580d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:05.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.105+0000 7f1765d74640 1 -- 172.21.15.157:0/2401009331 shutdown_connections 2024-03-29T11:13:05.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.105+0000 7f1765d74640 1 -- 172.21.15.157:0/2401009331 >> 172.21.15.157:0/2401009331 conn(0x7f1758005c00 msgr2=0x7f1758006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:05.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.106+0000 7f1765d74640 1 -- 172.21.15.157:0/2401009331 shutdown_connections 2024-03-29T11:13:05.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.106+0000 7f1765d74640 1 -- 172.21.15.157:0/2401009331 wait complete. 2024-03-29T11:13:05.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.107+0000 7f1765d74640 10 monclient: build_initial_monmap 2024-03-29T11:13:05.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.107+0000 7f1765d74640 10 monclient: monmap: 2024-03-29T11:13:05.086 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:05.086 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:05.086 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:13:05.087 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:13:05.087 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:05.087 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:05.087 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:05.087 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:13:05.087 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:13:05.087 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:05.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.107+0000 7f1765d74640 1 librados: starting msgr at 2024-03-29T11:13:05.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.107+0000 7f1765d74640 1 librados: starting objecter 2024-03-29T11:13:05.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.107+0000 7f1765d74640 1 Processor -- start 2024-03-29T11:13:05.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 1 -- start start 2024-03-29T11:13:05.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 1 librados: setting wanted keys 2024-03-29T11:13:05.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 1 librados: calling monclient init 2024-03-29T11:13:05.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 10 monclient: init 2024-03-29T11:13:05.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:05.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:05.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 10 monclient(hunting): picked mon.noname-c con 0x7f1758006e30 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:05.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 10 monclient(hunting): picked mon.noname-a con 0x7f1758035e00 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:05.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 10 monclient(hunting): picked mon.noname-b con 0x7f17580c9670 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:05.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1758033c00 con 0x7f1758035e00 2024-03-29T11:13:05.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1758160e10 con 0x7f17580c9670 2024-03-29T11:13:05.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1758033e30 con 0x7f1758006e30 2024-03-29T11:13:05.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:05.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.108+0000 7f1765d74640 10 monclient(hunting): authenticate will time out at 997.305849s 2024-03-29T11:13:05.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.112+0000 7f175ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1758035e00 0x7f17581612a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:52401/0 (socket says 172.21.15.157:52401) 2024-03-29T11:13:05.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.112+0000 7f175ffff640 1 -- 172.21.15.157:0/455644748 learned_addr learned my addr 172.21.15.157:0/455644748 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:05.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 10 client.?.objecter ms_handle_connect 0x7f1758035e00 2024-03-29T11:13:05.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:05.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965442615 0 0) 0x7f1758033c00 con 0x7f1758035e00 2024-03-29T11:13:05.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:05.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:05.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:05.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:13:05.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1744004eb0 con 0x7f1758035e00 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 188102148 0 0) 0x7f1744004eb0 con 0x7f1758035e00 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1758033c00 con 0x7f1758035e00 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f176004fdd0 con 0x7f1758035e00 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 10 monclient(hunting): dump: 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:05.094 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:05.095 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:05.095 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:05.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.115+0000 7f175d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:05.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4093087240 0 0) 0x7f1758033c00 con 0x7f1758035e00 2024-03-29T11:13:05.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 >> v1:172.21.15.157:6790/0 conn(0x7f1758006e30 legacy=0x7f17580325c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:05.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 >> v1:172.21.15.195:6789/0 conn(0x7f17580c9670 legacy=0x7f175802f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:05.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:13:05.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 1 monclient: found mon.a 2024-03-29T11:13:05.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:05.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:05.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17580ca000 con 0x7f1758035e00 2024-03-29T11:13:05.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f175d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f1765d74640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f1765d74640 10 monclient: _renew_subs 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f1765d74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f1765d74640 1 -- 172.21.15.157:0/455644748 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17581619b0 con 0x7f1758035e00 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f1765d74640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f1765d74640 10 monclient: _renew_subs 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f1765d74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f1765d74640 1 -- 172.21.15.157:0/455644748 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1758030100 con 0x7f1758035e00 2024-03-29T11:13:05.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.116+0000 7f1765d74640 1 librados: init done 2024-03-29T11:13:05.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.118+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f176004f400 con 0x7f1758035e00 2024-03-29T11:13:05.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.118+0000 7f175d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:05.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.118+0000 7f175effd640 4 set_mon_vals no callback set 2024-03-29T11:13:05.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.119+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1760075f10 con 0x7f1758035e00 2024-03-29T11:13:05.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.119+0000 7f175d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:05.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.119+0000 7f175d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:05.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.119+0000 7f175d7fa640 10 monclient: dump: 2024-03-29T11:13:05.100 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:05.101 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:05.101 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:05.101 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:05.101 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:05.101 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:05.101 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:05.101 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:05.101 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.119+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f1760067bf0 con 0x7f1758035e00 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.120+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f17600ff7b0 con 0x7f1758035e00 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.120+0000 7f175d7fa640 10 client.4293.objecter ms_dispatch 0x7f1760000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.120+0000 7f175d7fa640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.120+0000 7f175d7fa640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.120+0000 7f175d7fa640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.120+0000 7f1765d74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.120+0000 7f1765d74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.120+0000 7f1765d74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.120+0000 7f1765d74640 1 -- 172.21.15.157:0/455644748 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1740005080 con 0x7f1758035e00 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.121+0000 7f1764d72640 10 monclient: get_auth_request con 0x7f174406dcd0 auth_method 0 2024-03-29T11:13:05.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.122+0000 7f175d7fa640 10 client.4293.objecter ms_handle_connect 0x7f174406dcd0 2024-03-29T11:13:05.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.129+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17600cb110 con 0x7f1758035e00 2024-03-29T11:13:05.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.129+0000 7f175d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:05.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.129+0000 7f175d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:05.438 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.460+0000 7f1765d74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:05.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.460+0000 7f1765d74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:05.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.460+0000 7f1765d74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:05.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.460+0000 7f1765d74640 1 -- 172.21.15.157:0/455644748 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1740005330 con 0x7f1758035e00 2024-03-29T11:13:05.442 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.463+0000 7f175d7fa640 1 -- 172.21.15.157:0/455644748 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f1760067f20 con 0x7f1758035e00 2024-03-29T11:13:05.442 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.463+0000 7f175d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:05.442 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.463+0000 7f175d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:05.442 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:13:05.443 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:05.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.467+0000 7f1765d74640 10 librados: watch_flush enter 2024-03-29T11:13:05.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.467+0000 7f1765d74640 10 librados: watch_flush exit 2024-03-29T11:13:05.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.467+0000 7f1765d74640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-29T11:13:05.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.467+0000 7f1765d74640 10 client.4293.objecter successfully canceled tick 2024-03-29T11:13:05.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.467+0000 7f1765d74640 1 -- 172.21.15.157:0/455644748 >> v1:172.21.15.157:6816/1747058247 conn(0x7f174406dcd0 legacy=0x7f1744070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:05.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.467+0000 7f1765d74640 10 monclient: shutdown 2024-03-29T11:13:05.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.467+0000 7f1765d74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:05.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.467+0000 7f1765d74640 1 -- 172.21.15.157:0/455644748 >> v1:172.21.15.157:6789/0 conn(0x7f1758035e00 legacy=0x7f17581612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:05.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.467+0000 7f1765d74640 1 -- 172.21.15.157:0/455644748 shutdown_connections 2024-03-29T11:13:05.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.467+0000 7f1765d74640 1 -- 172.21.15.157:0/455644748 >> 172.21.15.157:0/455644748 conn(0x7f1758005c00 msgr2=0x7f17580c4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:05.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.468+0000 7f1765d74640 1 -- 172.21.15.157:0/455644748 shutdown_connections 2024-03-29T11:13:05.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.468+0000 7f1765d74640 1 -- 172.21.15.157:0/455644748 wait complete. 2024-03-29T11:13:05.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:05.468+0000 7f1765d74640 1 librados: shutdown 2024-03-29T11:13:05.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:12.270 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:12.417 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.439+0000 7f85981d6640 10 monclient: get_monmap_and_config 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.439+0000 7f85981d6640 10 monclient: build_initial_monmap 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.439+0000 7f85981d6640 10 monclient: monmap: 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:13:12.440341+0000 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:13:12.440341+0000 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:13:12.418 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:12.419 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.441+0000 7f85981d6640 1 Processor -- start 2024-03-29T11:13:12.419 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.441+0000 7f85981d6640 1 -- start start 2024-03-29T11:13:12.419 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.441+0000 7f85981d6640 10 monclient: init 2024-03-29T11:13:12.420 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.442+0000 7f85981d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:12.420 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.442+0000 7f85981d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:12.420 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.442+0000 7f85981d6640 10 monclient(hunting): picked mon.noname-b con 0x7f85900a9ac0 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:12.420 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.442+0000 7f85981d6640 10 monclient(hunting): picked mon.noname-a con 0x7f859013ee90 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:12.420 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.442+0000 7f85981d6640 10 monclient(hunting): picked mon.noname-c con 0x7f85900af900 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:12.420 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.442+0000 7f85981d6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85900afd10 con 0x7f859013ee90 2024-03-29T11:13:12.420 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.442+0000 7f85981d6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85900af6f0 con 0x7f85900af900 2024-03-29T11:13:12.420 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.442+0000 7f85981d6640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85900aac80 con 0x7f85900a9ac0 2024-03-29T11:13:12.421 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.442+0000 7f85981d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:12.421 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.442+0000 7f85981d6640 10 monclient(hunting): authenticate will time out at 1004.639108s 2024-03-29T11:13:12.422 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.445+0000 7f8596773640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f85900af900 0x7f8590149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:38548/0 (socket says 172.21.15.157:38548) 2024-03-29T11:13:12.422 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.445+0000 7f8596773640 1 -- 172.21.15.157:0/1522391600 learned_addr learned my addr 172.21.15.157:0/1522391600 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:12.423 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.445+0000 7f8594f70640 1 -- 172.21.15.157:0/1522391600 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283791206 0 0) 0x7f85900af6f0 con 0x7f85900af900 2024-03-29T11:13:12.424 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient(hunting): _negotiate 2024-03-29T11:13:12.424 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:12.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:12.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:13:12.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 1 -- 172.21.15.157:0/1522391600 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8578003520 con 0x7f85900af900 2024-03-29T11:13:12.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 1 -- 172.21.15.157:0/1522391600 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1276536079 0 0) 0x7f8578003520 con 0x7f85900af900 2024-03-29T11:13:12.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 1 -- 172.21.15.157:0/1522391600 >> v1:172.21.15.157:6790/0 conn(0x7f85900a9ac0 legacy=0x7f85900adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:12.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 1 -- 172.21.15.157:0/1522391600 >> v1:172.21.15.157:6789/0 conn(0x7f859013ee90 legacy=0x7f859013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:12.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: _finish_hunting 0 2024-03-29T11:13:12.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 1 monclient: found mon.noname-c 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 1 -- 172.21.15.157:0/1522391600 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85900aaf60 con 0x7f85900af900 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: _finish_auth 0 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: _check_auth_tickets 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 1 -- 172.21.15.157:0/1522391600 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f8588001a10 con 0x7f85900af900 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: dump: 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:12.427 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f85981d6640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f85981d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 1 -- 172.21.15.157:0/1522391600 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8588001be0 con 0x7f85900af900 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 1 -- 172.21.15.157:0/1522391600 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f8588002030 con 0x7f85900af900 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.446+0000 7f8594f70640 10 monclient: dump: 2024-03-29T11:13:12.428 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.447+0000 7f85981d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.447+0000 7f85981d6640 4 set_mon_vals no callback set 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.447+0000 7f85981d6640 10 monclient: shutdown 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.447+0000 7f85981d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:12.429 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.447+0000 7f85981d6640 1 -- 172.21.15.157:0/1522391600 >> v1:172.21.15.195:6789/0 conn(0x7f85900af900 legacy=0x7f8590149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:12.432 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.449+0000 7f85981d6640 1 -- 172.21.15.157:0/1522391600 shutdown_connections 2024-03-29T11:13:12.432 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.449+0000 7f85981d6640 1 -- 172.21.15.157:0/1522391600 >> 172.21.15.157:0/1522391600 conn(0x7f85900b35e0 msgr2=0x7f85900b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:12.432 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.450+0000 7f85981d6640 1 -- 172.21.15.157:0/1522391600 shutdown_connections 2024-03-29T11:13:12.432 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.450+0000 7f85981d6640 1 -- 172.21.15.157:0/1522391600 wait complete. 2024-03-29T11:13:12.432 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.451+0000 7f85981d6640 10 monclient: build_initial_monmap 2024-03-29T11:13:12.432 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.451+0000 7f85981d6640 10 monclient: monmap: 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.451+0000 7f85981d6640 1 librados: starting msgr at 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.451+0000 7f85981d6640 1 librados: starting objecter 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.451+0000 7f85981d6640 1 Processor -- start 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.451+0000 7f85981d6640 1 -- start start 2024-03-29T11:13:12.433 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.451+0000 7f85981d6640 1 librados: setting wanted keys 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.451+0000 7f85981d6640 1 librados: calling monclient init 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.451+0000 7f85981d6640 10 monclient: init 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f85981d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f85981d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f85981d6640 10 monclient(hunting): picked mon.noname-a con 0x7f85900a9ac0 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f85981d6640 10 monclient(hunting): picked mon.noname-b con 0x7f85900af900 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f85981d6640 10 monclient(hunting): picked mon.noname-c con 0x7f859013ee90 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f85981d6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85900702a0 con 0x7f85900a9ac0 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f85981d6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8590142550 con 0x7f85900af900 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f85981d6640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859006ecf0 con 0x7f859013ee90 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f85981d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f85981d6640 10 monclient(hunting): authenticate will time out at 1004.649047s 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f8595771640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f85900af900 0x7f85901d6420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:39030/0 (socket says 172.21.15.157:39030) 2024-03-29T11:13:12.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f8595771640 1 -- 172.21.15.157:0/3585023870 learned_addr learned my addr 172.21.15.157:0/3585023870 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:12.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f858e7fc640 10 client.?.objecter ms_handle_connect 0x7f85900af900 2024-03-29T11:13:12.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.452+0000 7f858e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:12.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.455+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773690126 0 0) 0x7f8590142550 con 0x7f85900af900 2024-03-29T11:13:12.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.455+0000 7f858e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:12.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.455+0000 7f858e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:12.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.455+0000 7f858e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:12.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.455+0000 7f858e7fc640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:13:12.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.455+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f857c004eb0 con 0x7f85900af900 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.455+0000 7f858e7fc640 10 client.?.objecter ms_handle_connect 0x7f85900a9ac0 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.455+0000 7f858e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.455+0000 7f858e7fc640 10 client.?.objecter ms_handle_connect 0x7f859013ee90 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.455+0000 7f858e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896098375 0 0) 0x7f859006ecf0 con 0x7f859013ee90 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8590142550 con 0x7f859013ee90 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261087172 0 0) 0x7f85900702a0 con 0x7f85900a9ac0 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:12.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.456+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f859006ecf0 con 0x7f85900a9ac0 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2804275588 0 0) 0x7f857c004eb0 con 0x7f85900af900 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f857c005620 con 0x7f85900af900 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f8584003fe0 con 0x7f85900af900 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient(hunting): dump: 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:12.445 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4085448010 0 0) 0x7f8590142550 con 0x7f859013ee90 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f857c004eb0 con 0x7f859013ee90 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f8588001ef0 con 0x7f859013ee90 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient(hunting): dump: 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:12.446 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 692819777 0 0) 0x7f857c005620 con 0x7f85900af900 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 >> v1:172.21.15.157:6790/0 conn(0x7f859013ee90 legacy=0x7f85901d6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 >> v1:172.21.15.157:6789/0 conn(0x7f85900a9ac0 legacy=0x7f8590139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 monclient: found mon.b 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859006ee80 con 0x7f85900af900 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:12.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.457+0000 7f858e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f85981d6640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f85981d6640 10 monclient: _renew_subs 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f85981d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f85981d6640 1 -- 172.21.15.157:0/3585023870 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859006f240 con 0x7f85900af900 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f85981d6640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f85981d6640 10 monclient: _renew_subs 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f85981d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f85981d6640 1 -- 172.21.15.157:0/3585023870 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f859013f7a0 con 0x7f85900af900 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f85981d6640 1 librados: init done 2024-03-29T11:13:12.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8584000ed0 con 0x7f85900af900 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f858e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f8584001220 con 0x7f85900af900 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f858e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f858e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f858e7fc640 10 monclient: dump: 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:12.449 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.458+0000 7f858ffff640 4 set_mon_vals no callback set 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.460+0000 7f85981d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.460+0000 7f85981d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.460+0000 7f85981d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.460+0000 7f85981d6640 1 -- 172.21.15.157:0/3585023870 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8570005080 con 0x7f85900af900 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.461+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f8584001380 con 0x7f85900af900 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.462+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f85840547a0 con 0x7f85900af900 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.462+0000 7f858e7fc640 10 client.4288.objecter ms_dispatch 0x7f8590000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.462+0000 7f858e7fc640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.462+0000 7f858e7fc640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.462+0000 7f858e7fc640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.462+0000 7f8595f72640 10 monclient: get_auth_request con 0x7f857c06e100 auth_method 0 2024-03-29T11:13:12.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.463+0000 7f858e7fc640 10 client.4288.objecter ms_handle_connect 0x7f857c06e100 2024-03-29T11:13:12.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.473+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8584050a40 con 0x7f85900af900 2024-03-29T11:13:12.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.473+0000 7f858e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:12.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.473+0000 7f858e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:12.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.796+0000 7f85981d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:12.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.796+0000 7f85981d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:12.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.796+0000 7f85981d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:13:12.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.796+0000 7f85981d6640 1 -- 172.21.15.157:0/3585023870 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8570005330 con 0x7f85900af900 2024-03-29T11:13:12.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.797+0000 7f858e7fc640 1 -- 172.21.15.157:0/3585023870 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f8584050be0 con 0x7f85900af900 2024-03-29T11:13:12.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.797+0000 7f858e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:12.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.797+0000 7f858e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:12.781 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:13:12.781 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:12.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.801+0000 7f85981d6640 10 librados: watch_flush enter 2024-03-29T11:13:12.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.801+0000 7f85981d6640 10 librados: watch_flush exit 2024-03-29T11:13:12.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.801+0000 7f85981d6640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T11:13:12.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.801+0000 7f85981d6640 10 client.4288.objecter successfully canceled tick 2024-03-29T11:13:12.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.801+0000 7f85981d6640 1 -- 172.21.15.157:0/3585023870 >> v1:172.21.15.157:6816/1747058247 conn(0x7f857c06e100 legacy=0x7f857c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:12.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.801+0000 7f85981d6640 10 monclient: shutdown 2024-03-29T11:13:12.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.801+0000 7f85981d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:12.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.801+0000 7f85981d6640 1 -- 172.21.15.157:0/3585023870 >> v1:172.21.15.195:6789/0 conn(0x7f85900af900 legacy=0x7f85901d6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:12.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.801+0000 7f85981d6640 1 -- 172.21.15.157:0/3585023870 shutdown_connections 2024-03-29T11:13:12.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.801+0000 7f85981d6640 1 -- 172.21.15.157:0/3585023870 >> 172.21.15.157:0/3585023870 conn(0x7f85900b35e0 msgr2=0x7f859013b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:12.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.804+0000 7f85981d6640 1 -- 172.21.15.157:0/3585023870 shutdown_connections 2024-03-29T11:13:12.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.804+0000 7f85981d6640 1 -- 172.21.15.157:0/3585023870 wait complete. 2024-03-29T11:13:12.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:12.804+0000 7f85981d6640 1 librados: shutdown 2024-03-29T11:13:12.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:19.605 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:19.763 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.784+0000 7fcc5400f640 10 monclient: get_monmap_and_config 2024-03-29T11:13:19.763 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.784+0000 7fcc5400f640 10 monclient: build_initial_monmap 2024-03-29T11:13:19.763 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.784+0000 7fcc5400f640 10 monclient: monmap: 2024-03-29T11:13:19.763 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:19.763 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:19.763 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:13:19.785549+0000 2024-03-29T11:13:19.763 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:13:19.785549+0000 2024-03-29T11:13:19.763 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:19.763 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:19.763 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:19.764 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:13:19.764 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:13:19.764 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:19.764 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.785+0000 7fcc5400f640 1 Processor -- start 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.785+0000 7fcc5400f640 1 -- start start 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.785+0000 7fcc5400f640 10 monclient: init 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.786+0000 7fcc5400f640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.786+0000 7fcc5400f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.786+0000 7fcc5400f640 10 monclient(hunting): picked mon.noname-b con 0x7fcc4c0a9ac0 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.786+0000 7fcc5400f640 10 monclient(hunting): picked mon.noname-c con 0x7fcc4c13ee90 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.786+0000 7fcc5400f640 10 monclient(hunting): picked mon.noname-a con 0x7fcc4c0af900 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.786+0000 7fcc5400f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c0afd10 con 0x7fcc4c0af900 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.786+0000 7fcc5400f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c0af6f0 con 0x7fcc4c13ee90 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.786+0000 7fcc5400f640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c0aac80 con 0x7fcc4c0a9ac0 2024-03-29T11:13:19.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.786+0000 7fcc5400f640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:19.766 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.786+0000 7fcc5400f640 10 monclient(hunting): authenticate will time out at 1011.983649s 2024-03-29T11:13:19.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.790+0000 7fcc525ac640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fcc4c0af900 0x7fcc4c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:35057/0 (socket says 172.21.15.157:35057) 2024-03-29T11:13:19.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.790+0000 7fcc525ac640 1 -- 172.21.15.157:0/3815043803 learned_addr learned my addr 172.21.15.157:0/3815043803 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:19.769 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.791+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292455160 0 0) 0x7fcc4c0afd10 con 0x7fcc4c0af900 2024-03-29T11:13:19.769 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.791+0000 7fcc50da9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:19.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.791+0000 7fcc50da9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:19.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.791+0000 7fcc50da9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:19.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.791+0000 7fcc50da9640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:13:19.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.791+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc38003520 con 0x7fcc4c0af900 2024-03-29T11:13:19.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253204040 0 0) 0x7fcc4c0aac80 con 0x7fcc4c0a9ac0 2024-03-29T11:13:19.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:19.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:19.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:19.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4c0afd10 con 0x7fcc4c0a9ac0 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076346791 0 0) 0x7fcc4c0af6f0 con 0x7fcc4c13ee90 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4c0aac80 con 0x7fcc4c13ee90 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 994503205 0 0) 0x7fcc38003520 con 0x7fcc4c0af900 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 >> v1:172.21.15.157:6790/0 conn(0x7fcc4c0a9ac0 legacy=0x7fcc4c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 >> v1:172.21.15.195:6789/0 conn(0x7fcc4c13ee90 legacy=0x7fcc4c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient: _finish_hunting 0 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 1 monclient: found mon.noname-a 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:19.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c0aaf60 con 0x7fcc4c0af900 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient: _finish_auth 0 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient: _check_auth_tickets 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fcc3c001610 con 0x7fcc4c0af900 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.792+0000 7fcc50da9640 10 monclient: dump: 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:19.775 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.793+0000 7fcc5400f640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.793+0000 7fcc5400f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.794+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc3c001770 con 0x7fcc4c0af900 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.794+0000 7fcc50da9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.794+0000 7fcc50da9640 1 -- 172.21.15.157:0/3815043803 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fcc3c001bc0 con 0x7fcc4c0af900 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.794+0000 7fcc50da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.794+0000 7fcc50da9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.794+0000 7fcc50da9640 10 monclient: dump: 2024-03-29T11:13:19.776 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:19.777 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:19.777 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:19.777 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:19.777 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:19.777 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:19.777 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:19.777 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:19.778 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:19.778 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:19.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.794+0000 7fcc5400f640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:19.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.794+0000 7fcc5400f640 4 set_mon_vals no callback set 2024-03-29T11:13:19.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.795+0000 7fcc5400f640 10 monclient: shutdown 2024-03-29T11:13:19.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.795+0000 7fcc5400f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:19.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.795+0000 7fcc5400f640 1 -- 172.21.15.157:0/3815043803 >> v1:172.21.15.157:6789/0 conn(0x7fcc4c0af900 legacy=0x7fcc4c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:19.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.796+0000 7fcc5400f640 1 -- 172.21.15.157:0/3815043803 shutdown_connections 2024-03-29T11:13:19.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.796+0000 7fcc5400f640 1 -- 172.21.15.157:0/3815043803 >> 172.21.15.157:0/3815043803 conn(0x7fcc4c0b35e0 msgr2=0x7fcc4c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:19.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.797+0000 7fcc5400f640 1 -- 172.21.15.157:0/3815043803 shutdown_connections 2024-03-29T11:13:19.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.797+0000 7fcc5400f640 1 -- 172.21.15.157:0/3815043803 wait complete. 2024-03-29T11:13:19.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.800+0000 7fcc5400f640 10 monclient: build_initial_monmap 2024-03-29T11:13:19.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.800+0000 7fcc5400f640 10 monclient: monmap: 2024-03-29T11:13:19.779 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.801+0000 7fcc5400f640 1 librados: starting msgr at 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.801+0000 7fcc5400f640 1 librados: starting objecter 2024-03-29T11:13:19.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.801+0000 7fcc5400f640 1 Processor -- start 2024-03-29T11:13:19.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.801+0000 7fcc5400f640 1 -- start start 2024-03-29T11:13:19.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.801+0000 7fcc5400f640 1 librados: setting wanted keys 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.801+0000 7fcc5400f640 1 librados: calling monclient init 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.801+0000 7fcc5400f640 10 monclient: init 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.802+0000 7fcc5400f640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.802+0000 7fcc5400f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.802+0000 7fcc5400f640 10 monclient(hunting): picked mon.noname-b con 0x7fcc4c0a9ac0 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.802+0000 7fcc5400f640 10 monclient(hunting): picked mon.noname-c con 0x7fcc4c0af900 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.802+0000 7fcc5400f640 10 monclient(hunting): picked mon.noname-a con 0x7fcc4c13ee90 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.802+0000 7fcc5400f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c13def0 con 0x7fcc4c13ee90 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.802+0000 7fcc5400f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c13d760 con 0x7fcc4c0a9ac0 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.802+0000 7fcc5400f640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c13c0c0 con 0x7fcc4c0af900 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.802+0000 7fcc5400f640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:19.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.802+0000 7fcc5400f640 10 monclient(hunting): authenticate will time out at 1011.999323s 2024-03-29T11:13:19.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.808+0000 7fcc525ac640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fcc4c13ee90 0x7fcc4c1e7070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:35295/0 (socket says 172.21.15.157:35295) 2024-03-29T11:13:19.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.808+0000 7fcc525ac640 1 -- 172.21.15.157:0/1379374551 learned_addr learned my addr 172.21.15.157:0/1379374551 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:19.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.808+0000 7fcc427fc640 10 client.?.objecter ms_handle_connect 0x7fcc4c13ee90 2024-03-29T11:13:19.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.809+0000 7fcc427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:19.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.809+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733647462 0 0) 0x7fcc4c13def0 con 0x7fcc4c13ee90 2024-03-29T11:13:19.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.809+0000 7fcc427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:19.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.809+0000 7fcc427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:19.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.809+0000 7fcc427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:19.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.809+0000 7fcc427fc640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:13:19.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.809+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc30004eb0 con 0x7fcc4c13ee90 2024-03-29T11:13:19.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.813+0000 7fcc427fc640 10 client.?.objecter ms_handle_connect 0x7fcc4c0af900 2024-03-29T11:13:19.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.813+0000 7fcc427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:19.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.813+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2360586223 0 0) 0x7fcc30004eb0 con 0x7fcc4c13ee90 2024-03-29T11:13:19.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.813+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc4c13def0 con 0x7fcc4c13ee90 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.813+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fcc3c001a80 con 0x7fcc4c13ee90 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.813+0000 7fcc427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.813+0000 7fcc427fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.813+0000 7fcc427fc640 10 monclient(hunting): dump: 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:19.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.813+0000 7fcc427fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:19.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.814+0000 7fcc427fc640 10 client.?.objecter ms_handle_connect 0x7fcc4c0a9ac0 2024-03-29T11:13:19.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.814+0000 7fcc427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:19.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.814+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550886025 0 0) 0x7fcc4c13d760 con 0x7fcc4c0a9ac0 2024-03-29T11:13:19.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.814+0000 7fcc427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:19.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.814+0000 7fcc427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:19.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.814+0000 7fcc427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:19.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.814+0000 7fcc427fc640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:13:19.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.814+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc30004eb0 con 0x7fcc4c0a9ac0 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.815+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3776850538 0 0) 0x7fcc30004eb0 con 0x7fcc4c0a9ac0 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.815+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc4c13d760 con 0x7fcc4c0a9ac0 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.815+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fcc48001930 con 0x7fcc4c0a9ac0 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.815+0000 7fcc427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.815+0000 7fcc427fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.815+0000 7fcc427fc640 10 monclient(hunting): dump: 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:19.794 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:19.795 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:19.795 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:19.795 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:19.795 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:19.799 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808968028 0 0) 0x7fcc4c13c0c0 con 0x7fcc4c0af900 2024-03-29T11:13:19.799 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:19.799 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc30004eb0 con 0x7fcc4c0af900 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1726756126 0 0) 0x7fcc4c13def0 con 0x7fcc4c13ee90 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 >> v1:172.21.15.157:6790/0 conn(0x7fcc4c0af900 legacy=0x7fcc4c1346d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 >> v1:172.21.15.195:6789/0 conn(0x7fcc4c0a9ac0 legacy=0x7fcc4c133c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 1 monclient: found mon.a 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c0b31c0 con 0x7fcc4c13ee90 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:19.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.816+0000 7fcc427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc5400f640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc5400f640 10 monclient: _renew_subs 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc5400f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc5400f640 1 -- 172.21.15.157:0/1379374551 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c13ca80 con 0x7fcc4c13ee90 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc5400f640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc5400f640 10 monclient: _renew_subs 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc5400f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc5400f640 1 -- 172.21.15.157:0/1379374551 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c1df180 con 0x7fcc4c13ee90 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc5400f640 1 librados: init done 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc3c004a90 con 0x7fcc4c13ee90 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fcc3c001250 con 0x7fcc4c13ee90 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:19.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc427fc640 10 monclient: dump: 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.817+0000 7fcc43fff640 4 set_mon_vals no callback set 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.819+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fcc3c01ac10 con 0x7fcc4c13ee90 2024-03-29T11:13:19.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.820+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fcc3c084b00 con 0x7fcc4c13ee90 2024-03-29T11:13:19.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.820+0000 7fcc427fc640 10 client.4302.objecter ms_dispatch 0x7fcc4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:19.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.820+0000 7fcc427fc640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:19.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.820+0000 7fcc427fc640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:19.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.820+0000 7fcc427fc640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T11:13:19.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.821+0000 7fcc51dab640 10 monclient: get_auth_request con 0x7fcc3006e030 auth_method 0 2024-03-29T11:13:19.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.821+0000 7fcc427fc640 10 client.4302.objecter ms_handle_connect 0x7fcc3006e030 2024-03-29T11:13:19.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.821+0000 7fcc5400f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:19.804 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.821+0000 7fcc5400f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:19.804 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.821+0000 7fcc5400f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:19.804 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.821+0000 7fcc5400f640 1 -- 172.21.15.157:0/1379374551 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc34005080 con 0x7fcc4c13ee90 2024-03-29T11:13:19.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.833+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc3c050210 con 0x7fcc4c13ee90 2024-03-29T11:13:19.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.833+0000 7fcc427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:19.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:19.833+0000 7fcc427fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:20.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.152+0000 7fcc5400f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:20.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.152+0000 7fcc5400f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:20.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.152+0000 7fcc5400f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:20.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.152+0000 7fcc5400f640 1 -- 172.21.15.157:0/1379374551 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc34005330 con 0x7fcc4c13ee90 2024-03-29T11:13:20.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.154+0000 7fcc427fc640 1 -- 172.21.15.157:0/1379374551 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fcc3c052310 con 0x7fcc4c13ee90 2024-03-29T11:13:20.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.154+0000 7fcc427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:20.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.154+0000 7fcc427fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:20.133 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:13:20.134 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:20.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.157+0000 7fcc5400f640 10 librados: watch_flush enter 2024-03-29T11:13:20.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.157+0000 7fcc5400f640 10 librados: watch_flush exit 2024-03-29T11:13:20.136 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.157+0000 7fcc5400f640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T11:13:20.136 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.157+0000 7fcc5400f640 10 client.4302.objecter successfully canceled tick 2024-03-29T11:13:20.136 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.158+0000 7fcc5400f640 1 -- 172.21.15.157:0/1379374551 >> v1:172.21.15.157:6816/1747058247 conn(0x7fcc3006e030 legacy=0x7fcc300704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:20.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.158+0000 7fcc5400f640 10 monclient: shutdown 2024-03-29T11:13:20.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.158+0000 7fcc5400f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:20.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.158+0000 7fcc5400f640 1 -- 172.21.15.157:0/1379374551 >> v1:172.21.15.157:6789/0 conn(0x7fcc4c13ee90 legacy=0x7fcc4c1e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:20.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.160+0000 7fcc5400f640 1 -- 172.21.15.157:0/1379374551 shutdown_connections 2024-03-29T11:13:20.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.160+0000 7fcc5400f640 1 -- 172.21.15.157:0/1379374551 >> 172.21.15.157:0/1379374551 conn(0x7fcc4c0b35e0 msgr2=0x7fcc4c13b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:20.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.161+0000 7fcc5400f640 1 -- 172.21.15.157:0/1379374551 shutdown_connections 2024-03-29T11:13:20.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.161+0000 7fcc5400f640 1 -- 172.21.15.157:0/1379374551 wait complete. 2024-03-29T11:13:20.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:20.161+0000 7fcc5400f640 1 librados: shutdown 2024-03-29T11:13:20.157 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:26.970 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:27.124 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.145+0000 7fc0e6353640 10 monclient: get_monmap_and_config 2024-03-29T11:13:27.124 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.145+0000 7fc0e6353640 10 monclient: build_initial_monmap 2024-03-29T11:13:27.124 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.145+0000 7fc0e6353640 10 monclient: monmap: 2024-03-29T11:13:27.124 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:27.124 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:27.125 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:13:27.146958+0000 2024-03-29T11:13:27.125 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:13:27.146958+0000 2024-03-29T11:13:27.125 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:27.125 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:27.125 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:27.125 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:13:27.125 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:13:27.125 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:27.126 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.148+0000 7fc0e6353640 1 Processor -- start 2024-03-29T11:13:27.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.149+0000 7fc0e6353640 1 -- start start 2024-03-29T11:13:27.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.149+0000 7fc0e6353640 10 monclient: init 2024-03-29T11:13:27.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.150+0000 7fc0e6353640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:27.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.150+0000 7fc0e6353640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:27.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.150+0000 7fc0e6353640 10 monclient(hunting): picked mon.noname-a con 0x7fc0e00a9750 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:27.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.150+0000 7fc0e6353640 10 monclient(hunting): picked mon.noname-c con 0x7fc0e00ae220 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:27.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.153+0000 7fc0e5351640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc0e00a9750 0x7fc0e0141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:39939/0 (socket says 172.21.15.157:39939) 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.154+0000 7fc0e6353640 10 monclient(hunting): picked mon.noname-b con 0x7fc0e013b9d0 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.154+0000 7fc0e6353640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e0142490 con 0x7fc0e00a9750 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.154+0000 7fc0e6353640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e00ac130 con 0x7fc0e00ae220 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.154+0000 7fc0e5b52640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fc0e013b9d0 0x7fc0e0149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:40357/0 (socket says 172.21.15.157:40357) 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.154+0000 7fc0e5b52640 1 -- 172.21.15.157:0/3421141587 learned_addr learned my addr 172.21.15.157:0/3421141587 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.154+0000 7fc0e6353640 1 -- 172.21.15.157:0/3421141587 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e00ac410 con 0x7fc0e013b9d0 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.154+0000 7fc0e6353640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.154+0000 7fc0e6353640 10 monclient(hunting): authenticate will time out at 1019.351166s 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391953909 0 0) 0x7fc0e0142490 con 0x7fc0e00a9750 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:27.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:27.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:27.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:13:27.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0cc003520 con 0x7fc0e00a9750 2024-03-29T11:13:27.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194025482 0 0) 0x7fc0e00ac410 con 0x7fc0e013b9d0 2024-03-29T11:13:27.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:27.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:27.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:27.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:13:27.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.155+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0e0142490 con 0x7fc0e013b9d0 2024-03-29T11:13:27.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.156+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194271130 0 0) 0x7fc0e00ac130 con 0x7fc0e00ae220 2024-03-29T11:13:27.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.156+0000 7fc0d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:27.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.156+0000 7fc0d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:27.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.156+0000 7fc0d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:27.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.156+0000 7fc0d7fff640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:13:27.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.156+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0e00ac410 con 0x7fc0e00ae220 2024-03-29T11:13:27.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1389860721 0 0) 0x7fc0e00ac410 con 0x7fc0e00ae220 2024-03-29T11:13:27.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 >> v1:172.21.15.157:6790/0 conn(0x7fc0e013b9d0 legacy=0x7fc0e0149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:27.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 >> v1:172.21.15.157:6789/0 conn(0x7fc0e00a9750 legacy=0x7fc0e0141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:27.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:13:27.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 1 monclient: found mon.noname-c 2024-03-29T11:13:27.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:27.140 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:13:27.140 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e00afa40 con 0x7fc0e00ae220 2024-03-29T11:13:27.140 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 10 monclient: _finish_auth 0 2024-03-29T11:13:27.140 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:13:27.140 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:27.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc0dc0015b0 con 0x7fc0e00ae220 2024-03-29T11:13:27.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:27.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:27.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0d7fff640 10 monclient: dump: 2024-03-29T11:13:27.141 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0e6353640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:13:27.142 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.160+0000 7fc0e6353640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.161+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0dc001710 con 0x7fc0e00ae220 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.161+0000 7fc0d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.161+0000 7fc0d7fff640 1 -- 172.21.15.157:0/3421141587 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc0dc001b60 con 0x7fc0e00ae220 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.161+0000 7fc0e6353640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.161+0000 7fc0e6353640 4 set_mon_vals no callback set 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.162+0000 7fc0e6353640 10 monclient: shutdown 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.161+0000 7fc0d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.162+0000 7fc0d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.162+0000 7fc0d7fff640 10 monclient: dump: 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:27.145 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.162+0000 7fc0e6353640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.162+0000 7fc0e6353640 1 -- 172.21.15.157:0/3421141587 >> v1:172.21.15.195:6789/0 conn(0x7fc0e00ae220 legacy=0x7fc0e013b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.164+0000 7fc0e6353640 1 -- 172.21.15.157:0/3421141587 shutdown_connections 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.164+0000 7fc0e6353640 1 -- 172.21.15.157:0/3421141587 >> 172.21.15.157:0/3421141587 conn(0x7fc0e00673e0 msgr2=0x7fc0e00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:27.146 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.165+0000 7fc0e6353640 1 -- 172.21.15.157:0/3421141587 shutdown_connections 2024-03-29T11:13:27.147 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.165+0000 7fc0e6353640 1 -- 172.21.15.157:0/3421141587 wait complete. 2024-03-29T11:13:27.147 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.166+0000 7fc0e6353640 10 monclient: build_initial_monmap 2024-03-29T11:13:27.147 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.166+0000 7fc0e6353640 10 monclient: monmap: 2024-03-29T11:13:27.147 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:27.147 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:27.147 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:13:27.147 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:13:27.147 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:27.147 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:27.147 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:27.148 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:13:27.148 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:13:27.148 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:27.148 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.167+0000 7fc0e6353640 1 librados: starting msgr at 2024-03-29T11:13:27.148 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.167+0000 7fc0e6353640 1 librados: starting objecter 2024-03-29T11:13:27.148 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.167+0000 7fc0e6353640 1 Processor -- start 2024-03-29T11:13:27.153 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.170+0000 7fc0e6353640 1 -- start start 2024-03-29T11:13:27.153 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.170+0000 7fc0e6353640 1 librados: setting wanted keys 2024-03-29T11:13:27.153 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.170+0000 7fc0e6353640 1 librados: calling monclient init 2024-03-29T11:13:27.153 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.170+0000 7fc0e6353640 10 monclient: init 2024-03-29T11:13:27.153 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.171+0000 7fc0e6353640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:27.153 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.171+0000 7fc0e6353640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:27.154 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.171+0000 7fc0e6353640 10 monclient(hunting): picked mon.noname-a con 0x7fc0e00a9750 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:27.154 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.173+0000 7fc0e6353640 10 monclient(hunting): picked mon.noname-b con 0x7fc0e00ae220 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:27.154 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.173+0000 7fc0e6353640 10 monclient(hunting): picked mon.noname-c con 0x7fc0e013b9d0 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:27.154 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.173+0000 7fc0e6353640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e01409d0 con 0x7fc0e00a9750 2024-03-29T11:13:27.154 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.173+0000 7fc0e6353640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e01d2f50 con 0x7fc0e00ae220 2024-03-29T11:13:27.154 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.173+0000 7fc0e6353640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e013c390 con 0x7fc0e013b9d0 2024-03-29T11:13:27.154 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.173+0000 7fc0e6353640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:27.154 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.173+0000 7fc0e6353640 10 monclient(hunting): authenticate will time out at 1019.370431s 2024-03-29T11:13:27.154 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.175+0000 7fc0e4b50640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc0e00ae220 0x7fc0e01d33e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:37541/0 (socket says 172.21.15.157:37541) 2024-03-29T11:13:27.154 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.175+0000 7fc0e4b50640 1 -- 172.21.15.157:0/3931452096 learned_addr learned my addr 172.21.15.157:0/3931452096 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:27.155 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.175+0000 7fc0d5ffb640 10 client.?.objecter ms_handle_connect 0x7fc0e00ae220 2024-03-29T11:13:27.155 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.175+0000 7fc0d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:27.159 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.176+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110094392 0 0) 0x7fc0e01d2f50 con 0x7fc0e00ae220 2024-03-29T11:13:27.159 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.176+0000 7fc0d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:27.159 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.176+0000 7fc0d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:27.159 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.176+0000 7fc0d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:27.159 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.176+0000 7fc0d5ffb640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:13:27.159 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.176+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c4004eb0 con 0x7fc0e00ae220 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.176+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3828488750 0 0) 0x7fc0c4004eb0 con 0x7fc0e00ae220 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.176+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0e01d2f50 con 0x7fc0e00ae220 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.177+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc0dc001b20 con 0x7fc0e00ae220 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.177+0000 7fc0d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.177+0000 7fc0d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.177+0000 7fc0d5ffb640 10 monclient(hunting): dump: 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:27.160 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.177+0000 7fc0d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.177+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 705023507 0 0) 0x7fc0e01d2f50 con 0x7fc0e00ae220 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.177+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 >> v1:172.21.15.157:6790/0 conn(0x7fc0e013b9d0 legacy=0x7fc0e0132950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 >> v1:172.21.15.157:6789/0 conn(0x7fc0e00a9750 legacy=0x7fc0e01402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0d5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0d5ffb640 1 monclient: found mon.b 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0d5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e013f520 con 0x7fc0e00ae220 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0d5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0d5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0e6353640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0e6353640 10 monclient: _renew_subs 2024-03-29T11:13:27.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0e6353640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:13:27.162 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0e6353640 1 -- 172.21.15.157:0/3931452096 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e01d3af0 con 0x7fc0e00ae220 2024-03-29T11:13:27.162 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0e6353640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:27.162 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0e6353640 10 monclient: _renew_subs 2024-03-29T11:13:27.162 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0e6353640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:13:27.162 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0e6353640 1 -- 172.21.15.157:0/3931452096 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e00b1410 con 0x7fc0e00ae220 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.178+0000 7fc0e6353640 1 librados: init done 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.180+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0dc0038a0 con 0x7fc0e00ae220 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.180+0000 7fc0d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.180+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc0dc003cf0 con 0x7fc0e00ae220 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.180+0000 7fc0d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.180+0000 7fc0d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.180+0000 7fc0d5ffb640 10 monclient: dump: 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:27.163 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:27.164 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:27.164 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:27.164 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:27.164 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:27.164 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.183+0000 7fc0d6ffd640 4 set_mon_vals no callback set 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.184+0000 7fc0e6353640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.184+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fc0dc003e50 con 0x7fc0e00ae220 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.184+0000 7fc0e6353640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.184+0000 7fc0e6353640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.184+0000 7fc0e6353640 1 -- 172.21.15.157:0/3931452096 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c0005080 con 0x7fc0e00ae220 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.185+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fc0dc084c80 con 0x7fc0e00ae220 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.185+0000 7fc0d5ffb640 10 client.4300.objecter ms_dispatch 0x7fc0e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.185+0000 7fc0d5ffb640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.185+0000 7fc0d5ffb640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.185+0000 7fc0d5ffb640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.185+0000 7fc0e5351640 10 monclient: get_auth_request con 0x7fc0c406dad0 auth_method 0 2024-03-29T11:13:27.166 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.186+0000 7fc0d5ffb640 10 client.4300.objecter ms_handle_connect 0x7fc0c406dad0 2024-03-29T11:13:27.175 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.196+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0dc050560 con 0x7fc0e00ae220 2024-03-29T11:13:27.175 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.196+0000 7fc0d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:27.175 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.196+0000 7fc0d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:27.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.523+0000 7fc0e6353640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:27.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.523+0000 7fc0e6353640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:27.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.523+0000 7fc0e6353640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:13:27.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.523+0000 7fc0e6353640 1 -- 172.21.15.157:0/3931452096 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0c0005330 con 0x7fc0e00ae220 2024-03-29T11:13:27.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.529+0000 7fc0d5ffb640 1 -- 172.21.15.157:0/3931452096 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fc0dc052490 con 0x7fc0e00ae220 2024-03-29T11:13:27.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.529+0000 7fc0d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:27.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.529+0000 7fc0d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:27.508 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:13:27.508 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:27.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.533+0000 7fc0e6353640 10 librados: watch_flush enter 2024-03-29T11:13:27.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.533+0000 7fc0e6353640 10 librados: watch_flush exit 2024-03-29T11:13:27.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.533+0000 7fc0e6353640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T11:13:27.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.533+0000 7fc0e6353640 10 client.4300.objecter successfully canceled tick 2024-03-29T11:13:27.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.534+0000 7fc0e6353640 1 -- 172.21.15.157:0/3931452096 >> v1:172.21.15.157:6816/1747058247 conn(0x7fc0c406dad0 legacy=0x7fc0c406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:27.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.534+0000 7fc0e6353640 10 monclient: shutdown 2024-03-29T11:13:27.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.534+0000 7fc0e6353640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:27.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.534+0000 7fc0e6353640 1 -- 172.21.15.157:0/3931452096 >> v1:172.21.15.195:6789/0 conn(0x7fc0e00ae220 legacy=0x7fc0e01d33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:27.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.534+0000 7fc0e6353640 1 -- 172.21.15.157:0/3931452096 shutdown_connections 2024-03-29T11:13:27.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.534+0000 7fc0e6353640 1 -- 172.21.15.157:0/3931452096 >> 172.21.15.157:0/3931452096 conn(0x7fc0e00673e0 msgr2=0x7fc0e0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:27.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.535+0000 7fc0e6353640 1 -- 172.21.15.157:0/3931452096 shutdown_connections 2024-03-29T11:13:27.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.535+0000 7fc0e6353640 1 -- 172.21.15.157:0/3931452096 wait complete. 2024-03-29T11:13:27.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:27.535+0000 7fc0e6353640 1 librados: shutdown 2024-03-29T11:13:27.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:34.337 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:34.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.512+0000 7fb51598f640 10 monclient: get_monmap_and_config 2024-03-29T11:13:34.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.512+0000 7fb51598f640 10 monclient: build_initial_monmap 2024-03-29T11:13:34.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.512+0000 7fb51598f640 10 monclient: monmap: 2024-03-29T11:13:34.491 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:13:34.513512+0000 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:13:34.513512+0000 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.513+0000 7fb51598f640 1 Processor -- start 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.513+0000 7fb51598f640 1 -- start start 2024-03-29T11:13:34.492 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.513+0000 7fb51598f640 10 monclient: init 2024-03-29T11:13:34.493 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.514+0000 7fb51598f640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:34.493 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.514+0000 7fb51598f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:34.493 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.514+0000 7fb51598f640 10 monclient(hunting): picked mon.noname-b con 0x7fb5100a9ac0 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:34.493 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.514+0000 7fb51598f640 10 monclient(hunting): picked mon.noname-a con 0x7fb51013ee90 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:34.496 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.516+0000 7fb51598f640 10 monclient(hunting): picked mon.noname-c con 0x7fb5100af900 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:34.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.519+0000 7fb51598f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5100afd10 con 0x7fb51013ee90 2024-03-29T11:13:34.499 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.521+0000 7fb51598f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5100af6f0 con 0x7fb5100af900 2024-03-29T11:13:34.499 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.521+0000 7fb51598f640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5100aac80 con 0x7fb5100a9ac0 2024-03-29T11:13:34.499 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.521+0000 7fb51598f640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:34.499 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.521+0000 7fb51598f640 10 monclient(hunting): authenticate will time out at 1026.718167s 2024-03-29T11:13:34.500 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.521+0000 7fb50f7fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fb5100af900 0x7fb510149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:50289/0 (socket says 172.21.15.157:50289) 2024-03-29T11:13:34.500 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.521+0000 7fb50f7fe640 1 -- 172.21.15.157:0/3736693185 learned_addr learned my addr 172.21.15.157:0/3736693185 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:34.500 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.522+0000 7fb50dffb640 1 -- 172.21.15.157:0/3736693185 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302634385 0 0) 0x7fb5100af6f0 con 0x7fb5100af900 2024-03-29T11:13:34.500 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.522+0000 7fb50dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:34.500 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.522+0000 7fb50dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:34.500 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.522+0000 7fb50dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:34.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.522+0000 7fb50dffb640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:13:34.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.522+0000 7fb50dffb640 1 -- 172.21.15.157:0/3736693185 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f8003520 con 0x7fb5100af900 2024-03-29T11:13:34.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 1 -- 172.21.15.157:0/3736693185 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 829081659 0 0) 0x7fb4f8003520 con 0x7fb5100af900 2024-03-29T11:13:34.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 1 -- 172.21.15.157:0/3736693185 >> v1:172.21.15.157:6790/0 conn(0x7fb5100a9ac0 legacy=0x7fb5100adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:34.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 1 -- 172.21.15.157:0/3736693185 >> v1:172.21.15.157:6789/0 conn(0x7fb51013ee90 legacy=0x7fb51013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:34.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:34.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 1 monclient: found mon.noname-c 2024-03-29T11:13:34.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:34.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 1 -- 172.21.15.157:0/3736693185 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5100aaf60 con 0x7fb5100af900 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 1 -- 172.21.15.157:0/3736693185 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb508001a10 con 0x7fb5100af900 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.523+0000 7fb50dffb640 10 monclient: dump: 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:34.502 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:34.503 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:34.503 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:34.503 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:34.503 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:34.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.524+0000 7fb50dffb640 1 -- 172.21.15.157:0/3736693185 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb508001ca0 con 0x7fb5100af900 2024-03-29T11:13:34.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.524+0000 7fb50dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:34.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.524+0000 7fb50dffb640 1 -- 172.21.15.157:0/3736693185 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb508002030 con 0x7fb5100af900 2024-03-29T11:13:34.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.524+0000 7fb50dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:34.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.524+0000 7fb50dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.524+0000 7fb50dffb640 10 monclient: dump: 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.524+0000 7fb51598f640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.524+0000 7fb51598f640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.524+0000 7fb51598f640 4 set_mon_vals no callback set 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.525+0000 7fb51598f640 10 monclient: shutdown 2024-03-29T11:13:34.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.525+0000 7fb51598f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:34.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.525+0000 7fb51598f640 1 -- 172.21.15.157:0/3736693185 >> v1:172.21.15.195:6789/0 conn(0x7fb5100af900 legacy=0x7fb510149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:34.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.525+0000 7fb51598f640 1 -- 172.21.15.157:0/3736693185 shutdown_connections 2024-03-29T11:13:34.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.525+0000 7fb51598f640 1 -- 172.21.15.157:0/3736693185 >> 172.21.15.157:0/3736693185 conn(0x7fb5100b35e0 msgr2=0x7fb5100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:34.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.528+0000 7fb51598f640 1 -- 172.21.15.157:0/3736693185 shutdown_connections 2024-03-29T11:13:34.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.528+0000 7fb51598f640 1 -- 172.21.15.157:0/3736693185 wait complete. 2024-03-29T11:13:34.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.529+0000 7fb51598f640 10 monclient: build_initial_monmap 2024-03-29T11:13:34.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.529+0000 7fb51598f640 10 monclient: monmap: 2024-03-29T11:13:34.507 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:34.507 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:34.507 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:13:34.507 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:13:34.507 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:34.507 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:34.508 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:34.508 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:13:34.508 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:13:34.508 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:34.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.530+0000 7fb51598f640 1 librados: starting msgr at 2024-03-29T11:13:34.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.530+0000 7fb51598f640 1 librados: starting objecter 2024-03-29T11:13:34.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.530+0000 7fb51598f640 1 Processor -- start 2024-03-29T11:13:34.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.530+0000 7fb51598f640 1 -- start start 2024-03-29T11:13:34.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.530+0000 7fb51598f640 1 librados: setting wanted keys 2024-03-29T11:13:34.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.530+0000 7fb51598f640 1 librados: calling monclient init 2024-03-29T11:13:34.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.530+0000 7fb51598f640 10 monclient: init 2024-03-29T11:13:34.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.531+0000 7fb51598f640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:34.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.531+0000 7fb51598f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:13:34.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.531+0000 7fb51598f640 10 monclient(hunting): picked mon.noname-b con 0x7fb5100a9ac0 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:34.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.534+0000 7fb51598f640 10 monclient(hunting): picked mon.noname-a con 0x7fb5100af900 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:34.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.538+0000 7fb51598f640 10 monclient(hunting): picked mon.noname-c con 0x7fb51013ee90 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:34.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.538+0000 7fb51598f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb510140e90 con 0x7fb5100af900 2024-03-29T11:13:34.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.538+0000 7fb51598f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51013f9a0 con 0x7fb5100a9ac0 2024-03-29T11:13:34.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.538+0000 7fb51598f640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5101c9ad0 con 0x7fb51013ee90 2024-03-29T11:13:34.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.538+0000 7fb51598f640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:34.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.538+0000 7fb51598f640 10 monclient(hunting): authenticate will time out at 1026.734978s 2024-03-29T11:13:34.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.539+0000 7fb50e7fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb5100af900 0x7fb5101dbc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:60367/0 (socket says 172.21.15.157:60367) 2024-03-29T11:13:34.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.540+0000 7fb50e7fc640 1 -- 172.21.15.157:0/1517567953 learned_addr learned my addr 172.21.15.157:0/1517567953 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:34.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.540+0000 7fb4f7fff640 10 client.?.objecter ms_handle_connect 0x7fb5100af900 2024-03-29T11:13:34.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.540+0000 7fb4f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:34.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.540+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663234793 0 0) 0x7fb510140e90 con 0x7fb5100af900 2024-03-29T11:13:34.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.540+0000 7fb4f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:34.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.540+0000 7fb4f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:34.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.540+0000 7fb4f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:34.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.540+0000 7fb4f7fff640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:13:34.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.540+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4fc004eb0 con 0x7fb5100af900 2024-03-29T11:13:34.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3955810690 0 0) 0x7fb4fc004eb0 con 0x7fb5100af900 2024-03-29T11:13:34.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb510140e90 con 0x7fb5100af900 2024-03-29T11:13:34.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb504001990 con 0x7fb5100af900 2024-03-29T11:13:34.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:34.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:34.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 10 monclient(hunting): dump: 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 10 client.?.objecter ms_handle_connect 0x7fb5100a9ac0 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864801519 0 0) 0x7fb51013f9a0 con 0x7fb5100a9ac0 2024-03-29T11:13:34.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4fc004eb0 con 0x7fb5100a9ac0 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3573554466 0 0) 0x7fb510140e90 con 0x7fb5100af900 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 >> v1:172.21.15.157:6790/0 conn(0x7fb51013ee90 legacy=0x7fb5101dc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.541+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 >> v1:172.21.15.195:6789/0 conn(0x7fb5100a9ac0 legacy=0x7fb5101db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 1 monclient: found mon.a 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5101db820 con 0x7fb5100af900 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 10 monclient: _finish_auth 0 2024-03-29T11:13:34.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb51598f640 5 monclient: authenticate success, global_id 4311 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb51598f640 10 monclient: _renew_subs 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb51598f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb51598f640 1 -- 172.21.15.157:0/1517567953 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51013f5c0 con 0x7fb5100af900 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb504001d10 con 0x7fb5100af900 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb504003a60 con 0x7fb5100af900 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb4f7fff640 10 monclient: dump: 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:34.526 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb51598f640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb51598f640 10 monclient: _renew_subs 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb51598f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb51598f640 1 -- 172.21.15.157:0/1517567953 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5101c9210 con 0x7fb5100af900 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.542+0000 7fb51598f640 1 librados: init done 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.543+0000 7fb51598f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.543+0000 7fb51598f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.543+0000 7fb51598f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:34.527 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.543+0000 7fb51598f640 1 -- 172.21.15.157:0/1517567953 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51013f800 con 0x7fb5100af900 2024-03-29T11:13:34.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.543+0000 7fb50cff9640 4 set_mon_vals no callback set 2024-03-29T11:13:34.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.544+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fb504003bc0 con 0x7fb5100af900 2024-03-29T11:13:34.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.544+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fb504084bd0 con 0x7fb5100af900 2024-03-29T11:13:34.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.544+0000 7fb4f7fff640 10 client.4311.objecter ms_dispatch 0x7fb510000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:34.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.544+0000 7fb4f7fff640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:34.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.544+0000 7fb4f7fff640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:34.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.545+0000 7fb4f7fff640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T11:13:34.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.545+0000 7fb50effd640 10 monclient: get_auth_request con 0x7fb4fc06dd90 auth_method 0 2024-03-29T11:13:34.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.545+0000 7fb4f7fff640 10 client.4311.objecter ms_handle_connect 0x7fb4fc06dd90 2024-03-29T11:13:34.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.553+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb51013f800 con 0x7fb5100af900 2024-03-29T11:13:34.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.553+0000 7fb4f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:34.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.553+0000 7fb4f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:34.850 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.872+0000 7fb51598f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:34.850 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.872+0000 7fb51598f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:34.850 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.872+0000 7fb51598f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:34.850 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.872+0000 7fb51598f640 1 -- 172.21.15.157:0/1517567953 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb51013f800 con 0x7fb5100af900 2024-03-29T11:13:34.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.873+0000 7fb4f7fff640 1 -- 172.21.15.157:0/1517567953 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fb51013f800 con 0x7fb5100af900 2024-03-29T11:13:34.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.873+0000 7fb4f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:34.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.873+0000 7fb4f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:34.851 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:13:34.852 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:34.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.877+0000 7fb51598f640 10 librados: watch_flush enter 2024-03-29T11:13:34.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.877+0000 7fb51598f640 10 librados: watch_flush exit 2024-03-29T11:13:34.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.877+0000 7fb51598f640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T11:13:34.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.877+0000 7fb51598f640 10 client.4311.objecter successfully canceled tick 2024-03-29T11:13:34.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.878+0000 7fb51598f640 1 -- 172.21.15.157:0/1517567953 >> v1:172.21.15.157:6816/1747058247 conn(0x7fb4fc06dd90 legacy=0x7fb4fc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:34.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.878+0000 7fb51598f640 10 monclient: shutdown 2024-03-29T11:13:34.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.878+0000 7fb51598f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:34.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.878+0000 7fb51598f640 1 -- 172.21.15.157:0/1517567953 >> v1:172.21.15.157:6789/0 conn(0x7fb5100af900 legacy=0x7fb5101dbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:34.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.878+0000 7fb51598f640 1 -- 172.21.15.157:0/1517567953 shutdown_connections 2024-03-29T11:13:34.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.878+0000 7fb51598f640 1 -- 172.21.15.157:0/1517567953 >> 172.21.15.157:0/1517567953 conn(0x7fb5100b35e0 msgr2=0x7fb5100b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:34.858 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.880+0000 7fb51598f640 1 -- 172.21.15.157:0/1517567953 shutdown_connections 2024-03-29T11:13:34.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.881+0000 7fb51598f640 1 -- 172.21.15.157:0/1517567953 wait complete. 2024-03-29T11:13:34.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:34.881+0000 7fb51598f640 1 librados: shutdown 2024-03-29T11:13:34.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:41.686 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:41.838 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.859+0000 7f235ae84640 10 monclient: get_monmap_and_config 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.859+0000 7f235ae84640 10 monclient: build_initial_monmap 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.859+0000 7f235ae84640 10 monclient: monmap: 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:13:41.859915+0000 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:13:41.859915+0000 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:13:41.839 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:41.840 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.861+0000 7f235ae84640 1 Processor -- start 2024-03-29T11:13:41.840 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.861+0000 7f235ae84640 1 -- start start 2024-03-29T11:13:41.843 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.861+0000 7f235ae84640 10 monclient: init 2024-03-29T11:13:41.843 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.862+0000 7f235ae84640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:41.843 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.862+0000 7f235ae84640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:41.843 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.862+0000 7f235ae84640 10 monclient(hunting): picked mon.noname-b con 0x7f234c0389e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:41.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.862+0000 7f235ae84640 10 monclient(hunting): picked mon.noname-c con 0x7f234c038dc0 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:41.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.862+0000 7f235ae84640 10 monclient(hunting): picked mon.noname-a con 0x7f234c030660 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:41.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.862+0000 7f235ae84640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c0c54c0 con 0x7f234c030660 2024-03-29T11:13:41.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.862+0000 7f235ae84640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c036430 con 0x7f234c038dc0 2024-03-29T11:13:41.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.862+0000 7f235ae84640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c039840 con 0x7f234c0389e0 2024-03-29T11:13:41.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.862+0000 7f235ae84640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:41.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.862+0000 7f235ae84640 10 monclient(hunting): authenticate will time out at 1034.058761s 2024-03-29T11:13:41.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.867+0000 7f2359681640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f234c038dc0 0x7f234c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:39521/0 (socket says 172.21.15.157:39521) 2024-03-29T11:13:41.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.867+0000 7f2359681640 1 -- 172.21.15.157:0/2511800674 learned_addr learned my addr 172.21.15.157:0/2511800674 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:41.845 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.867+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796702328 0 0) 0x7f234c0c54c0 con 0x7f234c030660 2024-03-29T11:13:41.845 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.867+0000 7f2358e80640 10 monclient(hunting): _negotiate 2024-03-29T11:13:41.846 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.867+0000 7f2358e80640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:41.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.867+0000 7f2358e80640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:41.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.867+0000 7f2358e80640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:13:41.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.867+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2340003520 con 0x7f234c030660 2024-03-29T11:13:41.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762602284 0 0) 0x7f234c039840 con 0x7f234c0389e0 2024-03-29T11:13:41.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 10 monclient(hunting): _negotiate 2024-03-29T11:13:41.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c0c54c0 con 0x7f234c0389e0 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181134240 0 0) 0x7f234c036430 con 0x7f234c038dc0 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 10 monclient(hunting): _negotiate 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.868+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c039840 con 0x7f234c038dc0 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4128320895 0 0) 0x7f2340003520 con 0x7f234c030660 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 >> v1:172.21.15.157:6790/0 conn(0x7f234c0389e0 legacy=0x7f234c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:41.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 >> v1:172.21.15.195:6789/0 conn(0x7f234c038dc0 legacy=0x7f234c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: _finish_hunting 0 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 1 monclient: found mon.noname-a 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c035690 con 0x7f234c030660 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: _finish_auth 0 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: _check_auth_tickets 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2344001690 con 0x7f234c030660 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: dump: 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:41.856 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2344001870 con 0x7f234c030660 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 1 -- 172.21.15.157:0/2511800674 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2344001690 con 0x7f234c030660 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:41.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.869+0000 7f2358e80640 10 monclient: dump: 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.871+0000 7f235ae84640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.871+0000 7f235ae84640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:41.858 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.871+0000 7f235ae84640 4 set_mon_vals no callback set 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.872+0000 7f235ae84640 10 monclient: shutdown 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.872+0000 7f235ae84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.872+0000 7f235ae84640 1 -- 172.21.15.157:0/2511800674 >> v1:172.21.15.157:6789/0 conn(0x7f234c030660 legacy=0x7f234c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.873+0000 7f235ae84640 1 -- 172.21.15.157:0/2511800674 shutdown_connections 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.873+0000 7f235ae84640 1 -- 172.21.15.157:0/2511800674 >> 172.21.15.157:0/2511800674 conn(0x7f234c005c00 msgr2=0x7f234c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.875+0000 7f235ae84640 1 -- 172.21.15.157:0/2511800674 shutdown_connections 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.875+0000 7f235ae84640 1 -- 172.21.15.157:0/2511800674 wait complete. 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.876+0000 7f235ae84640 10 monclient: build_initial_monmap 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.876+0000 7f235ae84640 10 monclient: monmap: 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:41.859 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:41.860 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:41.860 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:13:41.860 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:13:41.860 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:41.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.876+0000 7f235ae84640 1 librados: starting msgr at 2024-03-29T11:13:41.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 1 librados: starting objecter 2024-03-29T11:13:41.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 1 Processor -- start 2024-03-29T11:13:41.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 1 -- start start 2024-03-29T11:13:41.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 1 librados: setting wanted keys 2024-03-29T11:13:41.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 1 librados: calling monclient init 2024-03-29T11:13:41.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 10 monclient: init 2024-03-29T11:13:41.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:41.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:41.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 10 monclient(hunting): picked mon.noname-a con 0x7f234c030660 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:41.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 10 monclient(hunting): picked mon.noname-c con 0x7f234c0389e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:41.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 10 monclient(hunting): picked mon.noname-b con 0x7f234c02fe10 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c032020 con 0x7f234c030660 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c0c0c60 con 0x7f234c02fe10 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c031cf0 con 0x7f234c0389e0 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.877+0000 7f235ae84640 10 monclient(hunting): authenticate will time out at 1034.073849s 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.880+0000 7f2359681640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f234c0389e0 0x7f234c0bfe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:58473/0 (socket says 172.21.15.157:58473) 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.880+0000 7f2359681640 1 -- 172.21.15.157:0/2648688694 learned_addr learned my addr 172.21.15.157:0/2648688694 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.881+0000 7f234a7fc640 10 client.?.objecter ms_handle_connect 0x7f234c0389e0 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.881+0000 7f234a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.881+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885640920 0 0) 0x7f234c031cf0 con 0x7f234c0389e0 2024-03-29T11:13:41.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.881+0000 7f234a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:41.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.881+0000 7f234a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:41.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.881+0000 7f234a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:41.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.881+0000 7f234a7fc640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:13:41.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.881+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2338004eb0 con 0x7f234c0389e0 2024-03-29T11:13:41.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.881+0000 7f234a7fc640 10 client.?.objecter ms_handle_connect 0x7f234c030660 2024-03-29T11:13:41.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.881+0000 7f234a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:41.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930246538 0 0) 0x7f234c032020 con 0x7f234c030660 2024-03-29T11:13:41.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c031cf0 con 0x7f234c030660 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2975072102 0 0) 0x7f2338004eb0 con 0x7f234c0389e0 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f234c032020 con 0x7f234c0389e0 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f235404fdd0 con 0x7f234c0389e0 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): dump: 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:41.873 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:41.874 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:41.875 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:41.875 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:41.875 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:41.875 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:41.875 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3116748284 0 0) 0x7f234c031cf0 con 0x7f234c030660 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2338004eb0 con 0x7f234c030660 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2350001930 con 0x7f234c030660 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 10 monclient(hunting): dump: 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:41.876 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3053677218 0 0) 0x7f2338004eb0 con 0x7f234c030660 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 >> v1:172.21.15.157:6790/0 conn(0x7f234c0389e0 legacy=0x7f234c0bfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.882+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 >> v1:172.21.15.195:6789/0 conn(0x7f234c02fe10 legacy=0x7f234c0c0550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f234a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f234a7fc640 1 monclient: found mon.a 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f234a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f234a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c0c58c0 con 0x7f234c030660 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f234a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f234a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f234a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:41.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f235ae84640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:13:41.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f235ae84640 10 monclient: _renew_subs 2024-03-29T11:13:41.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f235ae84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:41.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f235ae84640 1 -- 172.21.15.157:0/2648688694 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234c0c5b30 con 0x7f234c030660 2024-03-29T11:13:41.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f235ae84640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:41.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f235ae84640 10 monclient: _renew_subs 2024-03-29T11:13:41.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f235ae84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:41.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f235ae84640 1 -- 172.21.15.157:0/2648688694 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f234c02f3c0 con 0x7f234c030660 2024-03-29T11:13:41.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.884+0000 7f235ae84640 1 librados: init done 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.886+0000 7f235ae84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.886+0000 7f235ae84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.886+0000 7f235ae84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.886+0000 7f235ae84640 1 -- 172.21.15.157:0/2648688694 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234c007b90 con 0x7f234c030660 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.888+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23500035f0 con 0x7f234c030660 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.888+0000 7f234a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.888+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2350003900 con 0x7f234c030660 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.888+0000 7f234a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.888+0000 7f234a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.888+0000 7f234a7fc640 10 monclient: dump: 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:41.879 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.888+0000 7f234bfff640 4 set_mon_vals no callback set 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.889+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f2350019910 con 0x7f234c030660 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.890+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f2350083830 con 0x7f234c030660 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.890+0000 7f234a7fc640 10 client.4317.objecter ms_dispatch 0x7f2354000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.890+0000 7f234a7fc640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.890+0000 7f234a7fc640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:41.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.890+0000 7f234a7fc640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T11:13:41.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.893+0000 7f2359681640 10 monclient: get_auth_request con 0x7f233806dde0 auth_method 0 2024-03-29T11:13:41.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.893+0000 7f234a7fc640 10 client.4317.objecter ms_handle_connect 0x7f233806dde0 2024-03-29T11:13:41.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.902+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f235004f110 con 0x7f234c030660 2024-03-29T11:13:41.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.902+0000 7f234a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:41.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:41.902+0000 7f234a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:42.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.260+0000 7f235ae84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:42.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.260+0000 7f235ae84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:42.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.260+0000 7f235ae84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:42.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.260+0000 7f235ae84640 1 -- 172.21.15.157:0/2648688694 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f234c005120 con 0x7f234c030660 2024-03-29T11:13:42.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.262+0000 7f234a7fc640 1 -- 172.21.15.157:0/2648688694 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f2350051040 con 0x7f234c030660 2024-03-29T11:13:42.239 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:13:42.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.262+0000 7f234a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:42.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.262+0000 7f234a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:42.240 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.265+0000 7f235ae84640 10 librados: watch_flush enter 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 10 librados: watch_flush exit 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 10 client.4317.objecter successfully canceled tick 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 1 -- 172.21.15.157:0/2648688694 >> v1:172.21.15.157:6816/1747058247 conn(0x7f233806dde0 legacy=0x7f23380702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 10 monclient: shutdown 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 1 -- 172.21.15.157:0/2648688694 >> v1:172.21.15.157:6789/0 conn(0x7f234c030660 legacy=0x7f234c15c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 1 -- 172.21.15.157:0/2648688694 shutdown_connections 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 1 -- 172.21.15.157:0/2648688694 >> 172.21.15.157:0/2648688694 conn(0x7f234c005c00 msgr2=0x7f234c0c4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 1 -- 172.21.15.157:0/2648688694 shutdown_connections 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 1 -- 172.21.15.157:0/2648688694 wait complete. 2024-03-29T11:13:42.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:42.266+0000 7f235ae84640 1 librados: shutdown 2024-03-29T11:13:42.257 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:49.065 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:49.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.234+0000 7f9209725640 10 monclient: get_monmap_and_config 2024-03-29T11:13:49.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.234+0000 7f9209725640 10 monclient: build_initial_monmap 2024-03-29T11:13:49.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.234+0000 7f9209725640 10 monclient: monmap: 2024-03-29T11:13:49.213 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:49.213 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:49.213 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:13:49.235628+0000 2024-03-29T11:13:49.214 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:13:49.235628+0000 2024-03-29T11:13:49.214 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:49.214 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:49.214 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:49.214 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:13:49.214 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:13:49.214 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:49.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.236+0000 7f9209725640 1 Processor -- start 2024-03-29T11:13:49.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.236+0000 7f9209725640 1 -- start start 2024-03-29T11:13:49.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.236+0000 7f9209725640 10 monclient: init 2024-03-29T11:13:49.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.237+0000 7f9209725640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:49.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.237+0000 7f9209725640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:49.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.237+0000 7f9209725640 10 monclient(hunting): picked mon.noname-a con 0x7f92040a9930 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:49.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.237+0000 7f9209725640 10 monclient(hunting): picked mon.noname-b con 0x7f9204149dc0 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:49.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.237+0000 7f9209725640 10 monclient(hunting): picked mon.noname-c con 0x7f920414ff40 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:49.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.237+0000 7f9209725640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040aae60 con 0x7f92040a9930 2024-03-29T11:13:49.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.237+0000 7f9209725640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040b4310 con 0x7f920414ff40 2024-03-29T11:13:49.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.237+0000 7f9209725640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040afdd0 con 0x7f9204149dc0 2024-03-29T11:13:49.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.237+0000 7f9209725640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:49.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.237+0000 7f9209725640 10 monclient(hunting): authenticate will time out at 1041.434363s 2024-03-29T11:13:49.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.238+0000 7f92027fc640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f9204149dc0 0x7f920414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:48226/0 (socket says 172.21.15.157:48226) 2024-03-29T11:13:49.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.238+0000 7f92027fc640 1 -- 172.21.15.157:0/1212404110 learned_addr learned my addr 172.21.15.157:0/1212404110 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:49.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.240+0000 7f9201ffb640 1 -- 172.21.15.157:0/1212404110 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771870790 0 0) 0x7f92040afdd0 con 0x7f9204149dc0 2024-03-29T11:13:49.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.240+0000 7f9201ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:49.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.240+0000 7f9201ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:49.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.240+0000 7f9201ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:49.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.240+0000 7f9201ffb640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:13:49.218 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.240+0000 7f9201ffb640 1 -- 172.21.15.157:0/1212404110 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ec003520 con 0x7f9204149dc0 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 1 -- 172.21.15.157:0/1212404110 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 24167887 0 0) 0x7f91ec003520 con 0x7f9204149dc0 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 1 -- 172.21.15.157:0/1212404110 >> v1:172.21.15.195:6789/0 conn(0x7f920414ff40 legacy=0x7f9204152330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 1 -- 172.21.15.157:0/1212404110 >> v1:172.21.15.157:6789/0 conn(0x7f92040a9930 legacy=0x7f9204143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 1 monclient: found mon.noname-b 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 1 -- 172.21.15.157:0/1212404110 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920414a950 con 0x7f9204149dc0 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:49.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:49.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 1 -- 172.21.15.157:0/1212404110 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f91f4001690 con 0x7f9204149dc0 2024-03-29T11:13:49.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:49.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:49.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.241+0000 7f9201ffb640 10 monclient: dump: 2024-03-29T11:13:49.220 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:49.220 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:49.220 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.220 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.220 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:49.220 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:49.222 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:49.222 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:49.222 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:49.222 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:49.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.242+0000 7f9201ffb640 1 -- 172.21.15.157:0/1212404110 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91f4001860 con 0x7f9204149dc0 2024-03-29T11:13:49.226 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.242+0000 7f9201ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:49.226 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.242+0000 7f9201ffb640 1 -- 172.21.15.157:0/1212404110 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f91f4001cb0 con 0x7f9204149dc0 2024-03-29T11:13:49.226 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.242+0000 7f9201ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:49.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.242+0000 7f9201ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:49.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.242+0000 7f9201ffb640 10 monclient: dump: 2024-03-29T11:13:49.227 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:49.227 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:49.228 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.228 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.228 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:49.229 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:49.229 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:49.229 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:49.229 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:49.229 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:49.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.242+0000 7f9209725640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:13:49.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.242+0000 7f9209725640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:49.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.242+0000 7f9209725640 4 set_mon_vals no callback set 2024-03-29T11:13:49.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.243+0000 7f9209725640 10 monclient: shutdown 2024-03-29T11:13:49.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.243+0000 7f9209725640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:49.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.243+0000 7f9209725640 1 -- 172.21.15.157:0/1212404110 >> v1:172.21.15.157:6790/0 conn(0x7f9204149dc0 legacy=0x7f920414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:49.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.244+0000 7f9209725640 1 -- 172.21.15.157:0/1212404110 shutdown_connections 2024-03-29T11:13:49.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.244+0000 7f9209725640 1 -- 172.21.15.157:0/1212404110 >> 172.21.15.157:0/1212404110 conn(0x7f92040b35e0 msgr2=0x7f92040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:49.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.247+0000 7f9209725640 1 -- 172.21.15.157:0/1212404110 shutdown_connections 2024-03-29T11:13:49.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.247+0000 7f9209725640 1 -- 172.21.15.157:0/1212404110 wait complete. 2024-03-29T11:13:49.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.248+0000 7f9209725640 10 monclient: build_initial_monmap 2024-03-29T11:13:49.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.248+0000 7f9209725640 10 monclient: monmap: 2024-03-29T11:13:49.237 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:49.237 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:49.237 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:13:49.237 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:13:49.237 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.249+0000 7f9209725640 1 librados: starting msgr at 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.249+0000 7f9209725640 1 librados: starting objecter 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.249+0000 7f9209725640 1 Processor -- start 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.249+0000 7f9209725640 1 -- start start 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.249+0000 7f9209725640 1 librados: setting wanted keys 2024-03-29T11:13:49.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.249+0000 7f9209725640 1 librados: calling monclient init 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.249+0000 7f9209725640 10 monclient: init 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.250+0000 7f9209725640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.250+0000 7f9209725640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.250+0000 7f9209725640 10 monclient(hunting): picked mon.noname-b con 0x7f92040a9930 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.250+0000 7f9209725640 10 monclient(hunting): picked mon.noname-a con 0x7f9204149dc0 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.250+0000 7f9209725640 10 monclient(hunting): picked mon.noname-c con 0x7f920414ff40 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.250+0000 7f9209725640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920413dbc0 con 0x7f9204149dc0 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.250+0000 7f9209725640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920413dd50 con 0x7f92040a9930 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.250+0000 7f9209725640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9204146c80 con 0x7f920414ff40 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.250+0000 7f9209725640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:49.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.250+0000 7f9209725640 10 monclient(hunting): authenticate will time out at 1041.447290s 2024-03-29T11:13:49.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.253+0000 7f9202ffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f92040a9930 0x7f920413d4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:42351/0 (socket says 172.21.15.157:42351) 2024-03-29T11:13:49.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.253+0000 7f9202ffd640 1 -- 172.21.15.157:0/3556065126 learned_addr learned my addr 172.21.15.157:0/3556065126 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:49.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.254+0000 7f91eb7fe640 10 client.?.objecter ms_handle_connect 0x7f92040a9930 2024-03-29T11:13:49.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.254+0000 7f91eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:49.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.254+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068299646 0 0) 0x7f920413dd50 con 0x7f92040a9930 2024-03-29T11:13:49.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.254+0000 7f91eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:49.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.254+0000 7f91eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:49.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.254+0000 7f91eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:49.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.254+0000 7f91eb7fe640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:13:49.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.254+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f0004eb0 con 0x7f92040a9930 2024-03-29T11:13:49.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.255+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1388762993 0 0) 0x7f91f0004eb0 con 0x7f92040a9930 2024-03-29T11:13:49.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.255+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f920413dd50 con 0x7f92040a9930 2024-03-29T11:13:49.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.255+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f91f8001960 con 0x7f92040a9930 2024-03-29T11:13:49.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.255+0000 7f91eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:49.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.255+0000 7f91eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:49.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.255+0000 7f91eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:13:49.242 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:49.242 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:49.242 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.242 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.242 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:49.242 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:49.242 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:49.243 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:49.243 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:49.243 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:49.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.255+0000 7f91eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:49.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.255+0000 7f91eb7fe640 10 client.?.objecter ms_handle_connect 0x7f9204149dc0 2024-03-29T11:13:49.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.255+0000 7f91eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:49.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 10 client.?.objecter ms_handle_connect 0x7f920414ff40 2024-03-29T11:13:49.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:49.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664050636 0 0) 0x7f920413dbc0 con 0x7f9204149dc0 2024-03-29T11:13:49.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:49.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:49.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:49.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:13:49.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f0004eb0 con 0x7f9204149dc0 2024-03-29T11:13:49.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802195727 0 0) 0x7f9204146c80 con 0x7f920414ff40 2024-03-29T11:13:49.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:49.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:49.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:49.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:13:49.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.256+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f00058b0 con 0x7f920414ff40 2024-03-29T11:13:49.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1407607580 0 0) 0x7f91f0004eb0 con 0x7f9204149dc0 2024-03-29T11:13:49.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9204146c80 con 0x7f9204149dc0 2024-03-29T11:13:49.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f91f4001b70 con 0x7f9204149dc0 2024-03-29T11:13:49.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:49.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:49.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:13:49.246 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:49.246 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:49.246 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.246 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2625612565 0 0) 0x7f91f00058b0 con 0x7f920414ff40 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91f0006150 con 0x7f920414ff40 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f91fc0019d0 con 0x7f920414ff40 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:49.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:13:49.248 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:49.248 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:49.248 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.248 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.248 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:49.248 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:49.248 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:49.248 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:49.248 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:49.249 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:49.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1223076737 0 0) 0x7f9204146c80 con 0x7f9204149dc0 2024-03-29T11:13:49.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 >> v1:172.21.15.157:6790/0 conn(0x7f920414ff40 legacy=0x7f92041db8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:49.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 >> v1:172.21.15.195:6789/0 conn(0x7f92040a9930 legacy=0x7f920413d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:49.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:49.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 monclient: found mon.a 2024-03-29T11:13:49.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:49.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:49.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9204146e10 con 0x7f9204149dc0 2024-03-29T11:13:49.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:49.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:49.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:49.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91f4001d50 con 0x7f9204149dc0 2024-03-29T11:13:49.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:49.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f91f4001b70 con 0x7f9204149dc0 2024-03-29T11:13:49.251 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:49.251 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:49.251 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f91eb7fe640 10 monclient: dump: 2024-03-29T11:13:49.251 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:49.251 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:49.251 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.251 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:49.251 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:49.252 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:49.252 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:49.252 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:49.252 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:49.252 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:49.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.257+0000 7f9200ff9640 4 set_mon_vals no callback set 2024-03-29T11:13:49.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.259+0000 7f9209725640 5 monclient: authenticate success, global_id 4320 2024-03-29T11:13:49.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.259+0000 7f9209725640 10 monclient: _renew_subs 2024-03-29T11:13:49.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.259+0000 7f9209725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:49.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.259+0000 7f9209725640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9204146720 con 0x7f9204149dc0 2024-03-29T11:13:49.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.259+0000 7f9209725640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:49.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.259+0000 7f9209725640 10 monclient: _renew_subs 2024-03-29T11:13:49.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.259+0000 7f9209725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:49.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.259+0000 7f9209725640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9204148890 con 0x7f9204149dc0 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.259+0000 7f9209725640 1 librados: init done 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.261+0000 7f9209725640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.261+0000 7f9209725640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.261+0000 7f9209725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.261+0000 7f9209725640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92041db5d0 con 0x7f9204149dc0 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.263+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f91f401aa60 con 0x7f9204149dc0 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.264+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f91f40542c0 con 0x7f9204149dc0 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.264+0000 7f91eb7fe640 10 client.4320.objecter ms_dispatch 0x7f9204000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.264+0000 7f91eb7fe640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.264+0000 7f91eb7fe640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.264+0000 7f91eb7fe640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.264+0000 7f9202ffd640 10 monclient: get_auth_request con 0x7f91f006e6a0 auth_method 0 2024-03-29T11:13:49.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.265+0000 7f91eb7fe640 10 client.4320.objecter ms_handle_connect 0x7f91f006e6a0 2024-03-29T11:13:49.260 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.272+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91f4050560 con 0x7f9204149dc0 2024-03-29T11:13:49.260 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.272+0000 7f91eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:49.260 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.272+0000 7f91eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:49.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.592+0000 7f9209725640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:49.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.592+0000 7f9209725640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:49.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.592+0000 7f9209725640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:49.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.592+0000 7f9209725640 1 -- 172.21.15.157:0/3556065126 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92041db5d0 con 0x7f9204149dc0 2024-03-29T11:13:49.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.595+0000 7f91eb7fe640 1 -- 172.21.15.157:0/3556065126 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f91f4050560 con 0x7f9204149dc0 2024-03-29T11:13:49.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.595+0000 7f91eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:49.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.595+0000 7f91eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:49.579 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:13:49.579 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.599+0000 7f9209725640 10 librados: watch_flush enter 2024-03-29T11:13:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.599+0000 7f9209725640 10 librados: watch_flush exit 2024-03-29T11:13:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.599+0000 7f9209725640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-29T11:13:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.599+0000 7f9209725640 10 client.4320.objecter successfully canceled tick 2024-03-29T11:13:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.599+0000 7f9209725640 1 -- 172.21.15.157:0/3556065126 >> v1:172.21.15.157:6816/1747058247 conn(0x7f91f006e6a0 legacy=0x7f91f0070b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.599+0000 7f9209725640 10 monclient: shutdown 2024-03-29T11:13:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.599+0000 7f9209725640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.599+0000 7f9209725640 1 -- 172.21.15.157:0/3556065126 >> v1:172.21.15.157:6789/0 conn(0x7f9204149dc0 legacy=0x7f9204146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.600+0000 7f9209725640 1 -- 172.21.15.157:0/3556065126 shutdown_connections 2024-03-29T11:13:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.600+0000 7f9209725640 1 -- 172.21.15.157:0/3556065126 >> 172.21.15.157:0/3556065126 conn(0x7f92040b35e0 msgr2=0x7f92040b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.600+0000 7f9209725640 1 -- 172.21.15.157:0/3556065126 shutdown_connections 2024-03-29T11:13:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.600+0000 7f9209725640 1 -- 172.21.15.157:0/3556065126 wait complete. 2024-03-29T11:13:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:49.600+0000 7f9209725640 1 librados: shutdown 2024-03-29T11:13:49.595 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:56.402 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:56.557 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.578+0000 7f9642fa3640 10 monclient: get_monmap_and_config 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.578+0000 7f9642fa3640 10 monclient: build_initial_monmap 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.578+0000 7f9642fa3640 10 monclient: monmap: 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:13:56.580059+0000 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:13:56.580059+0000 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.580+0000 7f9642fa3640 1 Processor -- start 2024-03-29T11:13:56.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.580+0000 7f9642fa3640 1 -- start start 2024-03-29T11:13:56.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.580+0000 7f9642fa3640 10 monclient: init 2024-03-29T11:13:56.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9642fa3640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:56.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9642fa3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9642fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f963c1422b0 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9642fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f963c142690 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9642fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f963c0b17d0 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9642fa3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c142c80 con 0x7f963c1422b0 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9642fa3640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c0aae40 con 0x7f963c142690 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9642fa3640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c13fbd0 con 0x7f963c0b17d0 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9642fa3640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9642fa3640 10 monclient(hunting): authenticate will time out at 1048.778138s 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9640d3f640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f963c1422b0 0x7f963c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:39106/0 (socket says 172.21.15.157:39106) 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f9640d3f640 1 -- 172.21.15.157:0/3480802849 learned_addr learned my addr 172.21.15.157:0/3480802849 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816592049 0 0) 0x7f963c142c80 con 0x7f963c1422b0 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f963b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:56.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f963b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f963b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f963b7fe640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.581+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f962c003520 con 0x7f963c1422b0 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824421083 0 0) 0x7f963c13fbd0 con 0x7f963c0b17d0 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c142c80 con 0x7f963c0b17d0 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 449085779 0 0) 0x7f962c003520 con 0x7f963c1422b0 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 >> v1:172.21.15.157:6790/0 conn(0x7f963c0b17d0 legacy=0x7f963c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 >> v1:172.21.15.195:6789/0 conn(0x7f963c142690 legacy=0x7f963c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:56.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c151940 con 0x7f963c1422b0 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f9642fa3640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f9642fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9630001690 con 0x7f963c1422b0 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f963b7fe640 10 monclient: dump: 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:56.563 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:56.564 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:56.564 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:56.564 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:56.564 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:56.564 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:56.564 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:56.564 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:56.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.582+0000 7f9642fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:56.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96300017f0 con 0x7f963c1422b0 2024-03-29T11:13:56.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f963b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:56.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f963b7fe640 1 -- 172.21.15.157:0/3480802849 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9630001b80 con 0x7f963c1422b0 2024-03-29T11:13:56.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f963b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:56.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f963b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:56.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f963b7fe640 10 monclient: dump: 2024-03-29T11:13:56.565 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f9642fa3640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:56.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f9642fa3640 4 set_mon_vals no callback set 2024-03-29T11:13:56.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f9642fa3640 10 monclient: shutdown 2024-03-29T11:13:56.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f9642fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:56.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.583+0000 7f9642fa3640 1 -- 172.21.15.157:0/3480802849 >> v1:172.21.15.157:6789/0 conn(0x7f963c1422b0 legacy=0x7f963c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:56.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.584+0000 7f9642fa3640 1 -- 172.21.15.157:0/3480802849 shutdown_connections 2024-03-29T11:13:56.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.584+0000 7f9642fa3640 1 -- 172.21.15.157:0/3480802849 >> 172.21.15.157:0/3480802849 conn(0x7f963c066a00 msgr2=0x7f963c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:56.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.584+0000 7f9642fa3640 1 -- 172.21.15.157:0/3480802849 shutdown_connections 2024-03-29T11:13:56.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.584+0000 7f9642fa3640 1 -- 172.21.15.157:0/3480802849 wait complete. 2024-03-29T11:13:56.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.585+0000 7f9642fa3640 10 monclient: build_initial_monmap 2024-03-29T11:13:56.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.585+0000 7f9642fa3640 10 monclient: monmap: 2024-03-29T11:13:56.574 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:13:56.574 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:56.574 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:13:56.574 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:13:56.574 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:56.574 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:56.574 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:13:56.574 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:13:56.574 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:13:56.575 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:56.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.586+0000 7f9642fa3640 1 librados: starting msgr at 2024-03-29T11:13:56.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.586+0000 7f9642fa3640 1 librados: starting objecter 2024-03-29T11:13:56.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.586+0000 7f9642fa3640 1 Processor -- start 2024-03-29T11:13:56.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.586+0000 7f9642fa3640 1 -- start start 2024-03-29T11:13:56.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.586+0000 7f9642fa3640 1 librados: setting wanted keys 2024-03-29T11:13:56.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.586+0000 7f9642fa3640 1 librados: calling monclient init 2024-03-29T11:13:56.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.586+0000 7f9642fa3640 10 monclient: init 2024-03-29T11:13:56.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.587+0000 7f9642fa3640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:56.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.587+0000 7f9642fa3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:13:56.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.587+0000 7f9642fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f963c0b17d0 addr v1:172.21.15.195:6789/0 2024-03-29T11:13:56.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f9640d3f640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f963c0b17d0 0x7f963c12bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:47600/0 (socket says 172.21.15.157:47600) 2024-03-29T11:13:56.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f9640d3f640 1 -- 172.21.15.157:0/2515115702 learned_addr learned my addr 172.21.15.157:0/2515115702 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:13:56.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f9642fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f963c1422b0 addr v1:172.21.15.157:6789/0 2024-03-29T11:13:56.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f9642fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f963c142690 addr v1:172.21.15.157:6790/0 2024-03-29T11:13:56.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c06f360 con 0x7f963c1422b0 2024-03-29T11:13:56.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c12b930 con 0x7f963c0b17d0 2024-03-29T11:13:56.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c06f5b0 con 0x7f963c142690 2024-03-29T11:13:56.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f9642fa3640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:56.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f9642fa3640 10 monclient(hunting): authenticate will time out at 1048.785254s 2024-03-29T11:13:56.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f96397fa640 10 client.?.objecter ms_handle_connect 0x7f963c0b17d0 2024-03-29T11:13:56.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f96397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:56.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f96397fa640 10 client.?.objecter ms_handle_connect 0x7f963c1422b0 2024-03-29T11:13:56.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f96397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:56.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439090449 0 0) 0x7f963c06f360 con 0x7f963c1422b0 2024-03-29T11:13:56.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f96397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:56.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f96397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:56.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f96397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:56.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f96397fa640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:13:56.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.588+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9620004eb0 con 0x7f963c1422b0 2024-03-29T11:13:56.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 10 client.?.objecter ms_handle_connect 0x7f963c142690 2024-03-29T11:13:56.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:56.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3517597301 0 0) 0x7f9620004eb0 con 0x7f963c1422b0 2024-03-29T11:13:56.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f963c06f360 con 0x7f963c1422b0 2024-03-29T11:13:56.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f96280017d0 con 0x7f963c1422b0 2024-03-29T11:13:56.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:56.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:56.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 10 monclient(hunting): dump: 2024-03-29T11:13:56.579 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:56.579 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:56.579 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:56.579 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:56.580 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:56.580 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:56.580 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:56.580 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:56.580 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:56.580 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:56.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:56.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846263651 0 0) 0x7f963c06f5b0 con 0x7f963c142690 2024-03-29T11:13:56.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:56.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:56.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:56.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:13:56.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.589+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9620004eb0 con 0x7f963c142690 2024-03-29T11:13:56.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3100581562 0 0) 0x7f963c06f360 con 0x7f963c1422b0 2024-03-29T11:13:56.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 >> v1:172.21.15.157:6790/0 conn(0x7f963c142690 legacy=0x7f963c1dda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:56.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 >> v1:172.21.15.195:6789/0 conn(0x7f963c0b17d0 legacy=0x7f963c12bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:56.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 10 monclient: _finish_hunting 0 2024-03-29T11:13:56.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 1 monclient: found mon.a 2024-03-29T11:13:56.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:56.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:56.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c12daf0 con 0x7f963c1422b0 2024-03-29T11:13:56.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 10 monclient: _finish_auth 0 2024-03-29T11:13:56.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 10 monclient: _check_auth_tickets 2024-03-29T11:13:56.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:56.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f9642fa3640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:13:56.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f9642fa3640 10 monclient: _renew_subs 2024-03-29T11:13:56.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f9642fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:56.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963c12dce0 con 0x7f963c1422b0 2024-03-29T11:13:56.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f9642fa3640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:56.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f9642fa3640 10 monclient: _renew_subs 2024-03-29T11:13:56.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f9642fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:56.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f963c12e770 con 0x7f963c1422b0 2024-03-29T11:13:56.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f9642fa3640 1 librados: init done 2024-03-29T11:13:56.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9628001930 con 0x7f963c1422b0 2024-03-29T11:13:56.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f96397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:56.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.590+0000 7f963affd640 4 set_mon_vals no callback set 2024-03-29T11:13:56.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.591+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9628003790 con 0x7f963c1422b0 2024-03-29T11:13:56.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.591+0000 7f96397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:56.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.591+0000 7f96397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:56.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.591+0000 7f96397fa640 10 monclient: dump: 2024-03-29T11:13:56.586 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:13:56.586 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:13:56.586 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:56.587 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:13:56.587 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:56.587 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:13:56.587 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:13:56.587 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:13:56.587 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:13:56.587 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:13:56.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.592+0000 7f9642fa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:56.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.592+0000 7f9642fa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:56.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.592+0000 7f9642fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:56.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.592+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c005080 con 0x7f963c1422b0 2024-03-29T11:13:56.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.592+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f9628003970 con 0x7f963c1422b0 2024-03-29T11:13:56.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.593+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f96280837e0 con 0x7f963c1422b0 2024-03-29T11:13:56.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.593+0000 7f9640d3f640 10 monclient: get_auth_request con 0x7f962006ddc0 auth_method 0 2024-03-29T11:13:56.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.593+0000 7f96397fa640 10 client.4326.objecter ms_dispatch 0x7f963c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:56.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.593+0000 7f96397fa640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:56.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.593+0000 7f96397fa640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:56.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.594+0000 7f96397fa640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T11:13:56.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.594+0000 7f96397fa640 10 client.4326.objecter ms_handle_connect 0x7f962006ddc0 2024-03-29T11:13:56.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.601+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f962804f0c0 con 0x7f963c1422b0 2024-03-29T11:13:56.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.601+0000 7f96397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:56.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.601+0000 7f96397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:56.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.976+0000 7f9642fa3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:56.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.976+0000 7f9642fa3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:56.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.976+0000 7f9642fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:13:56.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.976+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f961c005330 con 0x7f963c1422b0 2024-03-29T11:13:56.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.977+0000 7f96397fa640 1 -- 172.21.15.157:0/2515115702 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f9628050ff0 con 0x7f963c1422b0 2024-03-29T11:13:56.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.977+0000 7f96397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:56.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.977+0000 7f96397fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:56.955 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:13:56.955 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:56.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 10 librados: watch_flush enter 2024-03-29T11:13:56.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 10 librados: watch_flush exit 2024-03-29T11:13:56.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T11:13:56.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 10 client.4326.objecter successfully canceled tick 2024-03-29T11:13:56.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 >> v1:172.21.15.157:6816/1747058247 conn(0x7f962006ddc0 legacy=0x7f9620070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:56.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 10 monclient: shutdown 2024-03-29T11:13:56.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:56.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 >> v1:172.21.15.157:6789/0 conn(0x7f963c1422b0 legacy=0x7f963c12c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:56.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 shutdown_connections 2024-03-29T11:13:56.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 >> 172.21.15.157:0/2515115702 conn(0x7f963c066a00 msgr2=0x7f963c12acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:56.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 shutdown_connections 2024-03-29T11:13:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 1 -- 172.21.15.157:0/2515115702 wait complete. 2024-03-29T11:13:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:13:56.981+0000 7f9642fa3640 1 librados: shutdown 2024-03-29T11:13:56.975 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:03.785 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:03.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.942+0000 7fd0e8457640 10 monclient: get_monmap_and_config 2024-03-29T11:14:03.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.942+0000 7fd0e8457640 10 monclient: build_initial_monmap 2024-03-29T11:14:03.922 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.942+0000 7fd0e8457640 10 monclient: monmap: 2024-03-29T11:14:03.922 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:03.922 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:03.922 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:14:03.943711+0000 2024-03-29T11:14:03.922 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:14:03.943711+0000 2024-03-29T11:14:03.922 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:03.923 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:03.923 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:03.923 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:14:03.923 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:14:03.923 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:03.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.943+0000 7fd0e8457640 1 Processor -- start 2024-03-29T11:14:03.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.943+0000 7fd0e8457640 1 -- start start 2024-03-29T11:14:03.924 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.943+0000 7fd0e8457640 10 monclient: init 2024-03-29T11:14:03.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.944+0000 7fd0e8457640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:03.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.944+0000 7fd0e8457640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:03.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.944+0000 7fd0e8457640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e006ab70 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:03.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.944+0000 7fd0e8457640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e006af50 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:03.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.944+0000 7fd0e8457640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e0141fb0 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:03.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.944+0000 7fd0e8457640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e00b0d50 con 0x7fd0e0141fb0 2024-03-29T11:14:03.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.944+0000 7fd0e8457640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e00b1030 con 0x7fd0e006af50 2024-03-29T11:14:03.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.944+0000 7fd0e8457640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e00afa70 con 0x7fd0e006ab70 2024-03-29T11:14:03.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.944+0000 7fd0e8457640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:03.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.944+0000 7fd0e8457640 10 monclient(hunting): authenticate will time out at 1056.141713s 2024-03-29T11:14:03.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e59f2640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd0e006af50 0x7fd0e0072290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:39160/0 (socket says 172.21.15.157:39160) 2024-03-29T11:14:03.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e59f2640 1 -- 172.21.15.157:0/287242839 learned_addr learned my addr 172.21.15.157:0/287242839 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:03.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797101897 0 0) 0x7fd0e00b1030 con 0x7fd0e006af50 2024-03-29T11:14:03.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 10 monclient(hunting): _negotiate 2024-03-29T11:14:03.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:03.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:03.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:14:03.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0cc003520 con 0x7fd0e006af50 2024-03-29T11:14:03.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076677933 0 0) 0x7fd0e00afa70 con 0x7fd0e006ab70 2024-03-29T11:14:03.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 10 monclient(hunting): _negotiate 2024-03-29T11:14:03.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:03.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:03.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:14:03.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e00b1030 con 0x7fd0e006ab70 2024-03-29T11:14:03.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2697020139 0 0) 0x7fd0cc003520 con 0x7fd0e006af50 2024-03-29T11:14:03.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 >> v1:172.21.15.157:6790/0 conn(0x7fd0e006ab70 legacy=0x7fd0e00af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:03.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.945+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 >> v1:172.21.15.157:6789/0 conn(0x7fd0e0141fb0 legacy=0x7fd0e00715b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:03.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e51f1640 10 monclient: _finish_hunting 0 2024-03-29T11:14:03.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e51f1640 1 monclient: found mon.noname-c 2024-03-29T11:14:03.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e51f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:03.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e51f1640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:14:03.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e01423e0 con 0x7fd0e006af50 2024-03-29T11:14:03.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e51f1640 10 monclient: _finish_auth 0 2024-03-29T11:14:03.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e51f1640 10 monclient: _check_auth_tickets 2024-03-29T11:14:03.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e51f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:03.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e8457640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:14:03.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e8457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:03.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd0d0001690 con 0x7fd0e006af50 2024-03-29T11:14:03.949 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.947+0000 7fd0e51f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:03.949 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e51f1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:03.949 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e51f1640 10 monclient: dump: 2024-03-29T11:14:03.949 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:03.949 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:03.949 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:03.949 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:03.949 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:03.949 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:03.950 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:03.950 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:03.950 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:03.950 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:03.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e8457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:03.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0d00018b0 con 0x7fd0e006af50 2024-03-29T11:14:03.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e51f1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:03.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e8457640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:03.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e8457640 4 set_mon_vals no callback set 2024-03-29T11:14:03.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e51f1640 1 -- 172.21.15.157:0/287242839 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd0d0001c40 con 0x7fd0e006af50 2024-03-29T11:14:03.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e51f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:03.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e51f1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:03.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.948+0000 7fd0e51f1640 10 monclient: dump: 2024-03-29T11:14:03.951 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:03.951 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:03.951 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.949+0000 7fd0e8457640 10 monclient: shutdown 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.949+0000 7fd0e8457640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.949+0000 7fd0e8457640 1 -- 172.21.15.157:0/287242839 >> v1:172.21.15.195:6789/0 conn(0x7fd0e006af50 legacy=0x7fd0e0072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:03.952 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.949+0000 7fd0e8457640 1 -- 172.21.15.157:0/287242839 shutdown_connections 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.949+0000 7fd0e8457640 1 -- 172.21.15.157:0/287242839 >> 172.21.15.157:0/287242839 conn(0x7fd0e0067470 msgr2=0x7fd0e00b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.949+0000 7fd0e8457640 1 -- 172.21.15.157:0/287242839 shutdown_connections 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.949+0000 7fd0e8457640 1 -- 172.21.15.157:0/287242839 wait complete. 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.950+0000 7fd0e8457640 10 monclient: build_initial_monmap 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.950+0000 7fd0e8457640 10 monclient: monmap: 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:14:03.953 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.951+0000 7fd0e8457640 1 librados: starting msgr at 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.951+0000 7fd0e8457640 1 librados: starting objecter 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.951+0000 7fd0e8457640 1 Processor -- start 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.951+0000 7fd0e8457640 1 -- start start 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.951+0000 7fd0e8457640 1 librados: setting wanted keys 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.951+0000 7fd0e8457640 1 librados: calling monclient init 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.951+0000 7fd0e8457640 10 monclient: init 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.952+0000 7fd0e8457640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.952+0000 7fd0e8457640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.952+0000 7fd0e8457640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e006ab70 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.952+0000 7fd0e8457640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e0141fb0 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.952+0000 7fd0e61f3640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fd0e006ab70 0x7fd0e0133bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:39870/0 (socket says 172.21.15.157:39870) 2024-03-29T11:14:03.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.952+0000 7fd0e61f3640 1 -- 172.21.15.157:0/3005619004 learned_addr learned my addr 172.21.15.157:0/3005619004 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.952+0000 7fd0e8457640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e01362d0 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e0132520 con 0x7fd0e0141fb0 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e0137360 con 0x7fd0e006ab70 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e0137640 con 0x7fd0e01362d0 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0e8457640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0e8457640 10 monclient(hunting): authenticate will time out at 1056.150027s 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 client.?.objecter ms_handle_connect 0x7fd0e006ab70 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 client.?.objecter ms_handle_connect 0x7fd0e01362d0 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536072109 0 0) 0x7fd0e0137640 con 0x7fd0e01362d0 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:03.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c4004eb0 con 0x7fd0e01362d0 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037493225 0 0) 0x7fd0e0137360 con 0x7fd0e006ab70 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.953+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e0137640 con 0x7fd0e006ab70 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1969789035 0 0) 0x7fd0c4004eb0 con 0x7fd0e01362d0 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0e0137360 con 0x7fd0e01362d0 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919626942 0 0) 0x7fd0e0137640 con 0x7fd0e006ab70 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0c4004eb0 con 0x7fd0e006ab70 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd0d8001a00 con 0x7fd0e01362d0 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:03.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd0dc001970 con 0x7fd0e006ab70 2024-03-29T11:14:03.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 446154283 0 0) 0x7fd0e0137360 con 0x7fd0e01362d0 2024-03-29T11:14:03.958 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 >> v1:172.21.15.195:6789/0 conn(0x7fd0e006ab70 legacy=0x7fd0e0133bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.954+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 >> v1:172.21.15.157:6789/0 conn(0x7fd0e0141fb0 legacy=0x7fd0e01ded10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 1 monclient: found mon.c 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e00b3630 con 0x7fd0e01362d0 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0e8457640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0e8457640 10 monclient: _renew_subs 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0e8457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e0138f60 con 0x7fd0e01362d0 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0e8457640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:03.959 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0e8457640 10 monclient: _renew_subs 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0e8457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e0137c40 con 0x7fd0e01362d0 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0e8457640 1 librados: init done 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0d8001b60 con 0x7fd0e01362d0 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd0d8003970 con 0x7fd0e01362d0 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0d6ffd640 10 monclient: dump: 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:03.960 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.956+0000 7fd0e49f0640 4 set_mon_vals no callback set 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.957+0000 7fd0e8457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.959+0000 7fd0e8457640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.959+0000 7fd0e8457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.959+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c8005080 con 0x7fd0e01362d0 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.959+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fd0d8019940 con 0x7fd0e01362d0 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.960+0000 7fd0e61f3640 10 monclient: get_auth_request con 0x7fd0c406df60 auth_method 0 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.960+0000 7fd0d6ffd640 10 client.4331.objecter ms_handle_connect 0x7fd0c406df60 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.960+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fd0d80839f0 con 0x7fd0e01362d0 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.960+0000 7fd0d6ffd640 10 client.4331.objecter ms_dispatch 0x7fd0e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:03.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.960+0000 7fd0d6ffd640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:03.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.960+0000 7fd0d6ffd640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:03.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.960+0000 7fd0d6ffd640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:14:03.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.968+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd0d804f100 con 0x7fd0e01362d0 2024-03-29T11:14:03.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.968+0000 7fd0d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:03.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:03.968+0000 7fd0d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:04.260 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.283+0000 7fd0e8457640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:04.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.283+0000 7fd0e8457640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:04.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.283+0000 7fd0e8457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:04.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.283+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0c8005330 con 0x7fd0e01362d0 2024-03-29T11:14:04.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.283+0000 7fd0d6ffd640 1 -- 172.21.15.157:0/3005619004 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fd0d8051200 con 0x7fd0e01362d0 2024-03-29T11:14:04.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.283+0000 7fd0d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:04.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.283+0000 7fd0d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:04.262 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:14:04.262 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:04.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 10 librados: watch_flush enter 2024-03-29T11:14:04.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 10 librados: watch_flush exit 2024-03-29T11:14:04.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:14:04.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:14:04.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 >> v1:172.21.15.157:6816/1747058247 conn(0x7fd0c406df60 legacy=0x7fd0c4070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:04.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 10 monclient: shutdown 2024-03-29T11:14:04.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:04.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 >> v1:172.21.15.157:6790/0 conn(0x7fd0e01362d0 legacy=0x7fd0e01e2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:04.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 shutdown_connections 2024-03-29T11:14:04.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 >> 172.21.15.157:0/3005619004 conn(0x7fd0e0067470 msgr2=0x7fd0e0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:04.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 shutdown_connections 2024-03-29T11:14:04.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 1 -- 172.21.15.157:0/3005619004 wait complete. 2024-03-29T11:14:04.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:04.287+0000 7fd0e8457640 1 librados: shutdown 2024-03-29T11:14:04.280 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:11.087 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:11.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.243+0000 7f09cd5bc640 10 monclient: get_monmap_and_config 2024-03-29T11:14:11.221 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.243+0000 7f09cd5bc640 10 monclient: build_initial_monmap 2024-03-29T11:14:11.221 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.243+0000 7f09cd5bc640 10 monclient: monmap: 2024-03-29T11:14:11.221 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:11.221 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:11.221 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:14:11.244155+0000 2024-03-29T11:14:11.221 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:14:11.244155+0000 2024-03-29T11:14:11.221 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:11.221 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:11.221 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:11.221 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:14:11.222 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:14:11.222 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:11.223 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.243+0000 7f09cd5bc640 1 Processor -- start 2024-03-29T11:14:11.223 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.243+0000 7f09cd5bc640 1 -- start start 2024-03-29T11:14:11.223 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.243+0000 7f09cd5bc640 10 monclient: init 2024-03-29T11:14:11.223 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09cd5bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:11.223 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09cd5bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:11.223 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09cd5bc640 10 monclient(hunting): picked mon.noname-c con 0x7f09c80b0500 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09cd5bc640 10 monclient(hunting): picked mon.noname-b con 0x7f09c80b08e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09cd5bc640 10 monclient(hunting): picked mon.noname-a con 0x7f09c80aeca0 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09cd5bc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c8140620 con 0x7f09c80aeca0 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09cd5bc640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c813daf0 con 0x7f09c80b0500 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09cd5bc640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c80ae180 con 0x7f09c80b08e0 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09cd5bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09cd5bc640 10 monclient(hunting): authenticate will time out at 1063.441536s 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09bffff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f09c80b08e0 0x7f09c813d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:40362/0 (socket says 172.21.15.157:40362) 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.244+0000 7f09bffff640 1 -- 172.21.15.157:0/2384529422 learned_addr learned my addr 172.21.15.157:0/2384529422 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.245+0000 7f09c67fc640 1 -- 172.21.15.157:0/2384529422 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267185054 0 0) 0x7f09c80ae180 con 0x7f09c80b08e0 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.245+0000 7f09c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:11.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.245+0000 7f09c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:11.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.245+0000 7f09c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:11.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.245+0000 7f09c67fc640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:14:11.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.245+0000 7f09c67fc640 1 -- 172.21.15.157:0/2384529422 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b4003520 con 0x7f09c80b08e0 2024-03-29T11:14:11.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.245+0000 7f09c67fc640 1 -- 172.21.15.157:0/2384529422 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 941017934 0 0) 0x7f09b4003520 con 0x7f09c80b08e0 2024-03-29T11:14:11.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.245+0000 7f09c67fc640 1 -- 172.21.15.157:0/2384529422 >> v1:172.21.15.195:6789/0 conn(0x7f09c80b0500 legacy=0x7f09c813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:11.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.245+0000 7f09c67fc640 1 -- 172.21.15.157:0/2384529422 >> v1:172.21.15.157:6789/0 conn(0x7f09c80aeca0 legacy=0x7f09c814b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:11.226 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:11.226 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 1 monclient: found mon.noname-b 2024-03-29T11:14:11.226 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:11.226 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 1 -- 172.21.15.157:0/2384529422 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c80accf0 con 0x7f09c80b08e0 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 1 -- 172.21.15.157:0/2384529422 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f09b0001690 con 0x7f09c80b08e0 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: dump: 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:11.227 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09cd5bc640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09cd5bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 1 -- 172.21.15.157:0/2384529422 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09b0001860 con 0x7f09c80b08e0 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 1 -- 172.21.15.157:0/2384529422 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f09b0001cb0 con 0x7f09c80b08e0 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:11.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09c67fc640 10 monclient: dump: 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09cd5bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:11.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.248+0000 7f09cd5bc640 4 set_mon_vals no callback set 2024-03-29T11:14:11.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.249+0000 7f09cd5bc640 10 monclient: shutdown 2024-03-29T11:14:11.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.249+0000 7f09cd5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:11.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.249+0000 7f09cd5bc640 1 -- 172.21.15.157:0/2384529422 >> v1:172.21.15.157:6790/0 conn(0x7f09c80b08e0 legacy=0x7f09c813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:11.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.250+0000 7f09cd5bc640 1 -- 172.21.15.157:0/2384529422 shutdown_connections 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.250+0000 7f09cd5bc640 1 -- 172.21.15.157:0/2384529422 >> 172.21.15.157:0/2384529422 conn(0x7f09c806b660 msgr2=0x7f09c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.250+0000 7f09cd5bc640 1 -- 172.21.15.157:0/2384529422 shutdown_connections 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.250+0000 7f09cd5bc640 1 -- 172.21.15.157:0/2384529422 wait complete. 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.251+0000 7f09cd5bc640 10 monclient: build_initial_monmap 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.251+0000 7f09cd5bc640 10 monclient: monmap: 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:11.233 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:11.234 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:14:11.234 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:14:11.234 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:11.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.251+0000 7f09cd5bc640 1 librados: starting msgr at 2024-03-29T11:14:11.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.251+0000 7f09cd5bc640 1 librados: starting objecter 2024-03-29T11:14:11.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.252+0000 7f09cd5bc640 1 Processor -- start 2024-03-29T11:14:11.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.252+0000 7f09cd5bc640 1 -- start start 2024-03-29T11:14:11.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.252+0000 7f09cd5bc640 1 librados: setting wanted keys 2024-03-29T11:14:11.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.252+0000 7f09cd5bc640 1 librados: calling monclient init 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.252+0000 7f09cd5bc640 10 monclient: init 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09cd5bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09cd5bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09cd5bc640 10 monclient(hunting): picked mon.noname-b con 0x7f09c80aeca0 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09cd5bc640 10 monclient(hunting): picked mon.noname-c con 0x7f09c80b0500 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09cd5bc640 10 monclient(hunting): picked mon.noname-a con 0x7f09c813cb80 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09cd5bc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c811b5e0 con 0x7f09c813cb80 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09cd5bc640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c81c7030 con 0x7f09c80aeca0 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09cd5bc640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c813cfc0 con 0x7f09c80b0500 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09cd5bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:11.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09cd5bc640 10 monclient(hunting): authenticate will time out at 1063.449966s 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09bffff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f09c80b0500 0x7f09c81dbbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:41594/0 (socket says 172.21.15.157:41594) 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.253+0000 7f09bffff640 1 -- 172.21.15.157:0/1894483895 learned_addr learned my addr 172.21.15.157:0/1894483895 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 10 client.?.objecter ms_handle_connect 0x7f09c80b0500 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743909175 0 0) 0x7f09c813cfc0 con 0x7f09c80b0500 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a8004eb0 con 0x7f09c80b0500 2024-03-29T11:14:11.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2643016049 0 0) 0x7f09a8004eb0 con 0x7f09c80b0500 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09c813cfc0 con 0x7f09c80b0500 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f09b0001bb0 con 0x7f09c80b0500 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 10 monclient(hunting): dump: 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:11.237 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:11.238 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:11.238 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:11.238 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:11.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:11.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2303410697 0 0) 0x7f09c813cfc0 con 0x7f09c80b0500 2024-03-29T11:14:11.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 >> v1:172.21.15.195:6789/0 conn(0x7f09c80aeca0 legacy=0x7f09c813c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:11.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.254+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 >> v1:172.21.15.157:6789/0 conn(0x7f09c813cb80 legacy=0x7f09c81e7040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:11.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.255+0000 7f09bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:11.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.255+0000 7f09bf7fe640 1 monclient: found mon.c 2024-03-29T11:14:11.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.255+0000 7f09bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:11.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.255+0000 7f09bf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:11.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.255+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c811df70 con 0x7f09c80b0500 2024-03-29T11:14:11.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09bf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09cd5bc640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09cd5bc640 10 monclient: _renew_subs 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09cd5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09cd5bc640 1 -- 172.21.15.157:0/1894483895 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c81c6040 con 0x7f09c80b0500 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09cd5bc640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09cd5bc640 10 monclient: _renew_subs 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09cd5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09cd5bc640 1 -- 172.21.15.157:0/1894483895 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c811ee30 con 0x7f09c80b0500 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.256+0000 7f09cd5bc640 1 librados: init done 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.257+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09b0001d10 con 0x7f09c80b0500 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.257+0000 7f09cd5bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.257+0000 7f09cd5bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:11.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.257+0000 7f09cd5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.257+0000 7f09cd5bc640 1 -- 172.21.15.157:0/1894483895 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c80ad0c0 con 0x7f09c80b0500 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.258+0000 7f09bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.258+0000 7f09c5ffb640 4 set_mon_vals no callback set 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.258+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f09b0004c70 con 0x7f09c80b0500 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.258+0000 7f09bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.258+0000 7f09bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.258+0000 7f09bf7fe640 10 monclient: dump: 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:11.244 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:11.245 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:11.245 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:11.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.259+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f09b001ac40 con 0x7f09c80b0500 2024-03-29T11:14:11.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.260+0000 7f09c6ffd640 10 monclient: get_auth_request con 0x7f09a806dd10 auth_method 0 2024-03-29T11:14:11.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.260+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f09b0084b40 con 0x7f09c80b0500 2024-03-29T11:14:11.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.260+0000 7f09bf7fe640 10 client.4337.objecter ms_dispatch 0x7f09c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:11.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.260+0000 7f09bf7fe640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:11.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.260+0000 7f09bf7fe640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:11.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.260+0000 7f09bf7fe640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:14:11.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.260+0000 7f09bf7fe640 10 client.4337.objecter ms_handle_connect 0x7f09a806dd10 2024-03-29T11:14:11.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.266+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09b0050250 con 0x7f09c80b0500 2024-03-29T11:14:11.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.267+0000 7f09bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:11.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.267+0000 7f09bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:11.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.573+0000 7f09cd5bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:11.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.573+0000 7f09cd5bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:11.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.573+0000 7f09cd5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:11.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.573+0000 7f09cd5bc640 1 -- 172.21.15.157:0/1894483895 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09c8075f10 con 0x7f09c80b0500 2024-03-29T11:14:11.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.574+0000 7f09bf7fe640 1 -- 172.21.15.157:0/1894483895 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f09b0052350 con 0x7f09c80b0500 2024-03-29T11:14:11.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.574+0000 7f09bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:11.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.574+0000 7f09bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:11.553 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:14:11.553 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:11.555 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.577+0000 7f09cd5bc640 10 librados: watch_flush enter 2024-03-29T11:14:11.555 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.577+0000 7f09cd5bc640 10 librados: watch_flush exit 2024-03-29T11:14:11.555 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.577+0000 7f09cd5bc640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:14:11.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.577+0000 7f09cd5bc640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:14:11.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.578+0000 7f09cd5bc640 1 -- 172.21.15.157:0/1894483895 >> v1:172.21.15.157:6816/1747058247 conn(0x7f09a806dd10 legacy=0x7f09a80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:11.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.578+0000 7f09cd5bc640 10 monclient: shutdown 2024-03-29T11:14:11.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.578+0000 7f09cd5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:11.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.578+0000 7f09cd5bc640 1 -- 172.21.15.157:0/1894483895 >> v1:172.21.15.157:6790/0 conn(0x7f09c80b0500 legacy=0x7f09c81dbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:11.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.578+0000 7f09cd5bc640 1 -- 172.21.15.157:0/1894483895 shutdown_connections 2024-03-29T11:14:11.557 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.578+0000 7f09cd5bc640 1 -- 172.21.15.157:0/1894483895 >> 172.21.15.157:0/1894483895 conn(0x7f09c806b660 msgr2=0x7f09c80a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:11.557 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.578+0000 7f09cd5bc640 1 -- 172.21.15.157:0/1894483895 shutdown_connections 2024-03-29T11:14:11.557 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.578+0000 7f09cd5bc640 1 -- 172.21.15.157:0/1894483895 wait complete. 2024-03-29T11:14:11.557 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:11.578+0000 7f09cd5bc640 1 librados: shutdown 2024-03-29T11:14:11.571 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:18.378 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:18.530 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.551+0000 7f7d60255640 10 monclient: get_monmap_and_config 2024-03-29T11:14:18.530 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.551+0000 7f7d60255640 10 monclient: build_initial_monmap 2024-03-29T11:14:18.530 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.551+0000 7f7d60255640 10 monclient: monmap: 2024-03-29T11:14:18.531 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:18.531 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:18.531 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:14:18.552967+0000 2024-03-29T11:14:18.531 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:14:18.552967+0000 2024-03-29T11:14:18.531 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:18.531 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:18.531 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:18.531 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:14:18.531 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:14:18.532 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:18.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.552+0000 7f7d60255640 1 Processor -- start 2024-03-29T11:14:18.533 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.553+0000 7f7d60255640 1 -- start start 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.553+0000 7f7d60255640 10 monclient: init 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.553+0000 7f7d60255640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.553+0000 7f7d60255640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.553+0000 7f7d60255640 10 monclient(hunting): picked mon.noname-b con 0x7f7d581457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.553+0000 7f7d60255640 10 monclient(hunting): picked mon.noname-c con 0x7f7d5813f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.553+0000 7f7d60255640 10 monclient(hunting): picked mon.noname-a con 0x7f7d580a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.553+0000 7f7d60255640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d580aaef0 con 0x7f7d580a9d70 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.553+0000 7f7d60255640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d58145bd0 con 0x7f7d5813f650 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.554+0000 7f7d60255640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d58145eb0 con 0x7f7d581457f0 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.554+0000 7f7d60255640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:18.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.554+0000 7f7d60255640 10 monclient(hunting): authenticate will time out at 1070.750880s 2024-03-29T11:14:18.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.559+0000 7f7d5dff1640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f7d581457f0 0x7f7d5813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:52767/0 (socket says 172.21.15.157:52767) 2024-03-29T11:14:18.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.559+0000 7f7d5dff1640 1 -- 172.21.15.157:0/3297738595 learned_addr learned my addr 172.21.15.157:0/3297738595 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:18.539 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.560+0000 7f7d5cfef640 1 -- 172.21.15.157:0/3297738595 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637349580 0 0) 0x7f7d58145eb0 con 0x7f7d581457f0 2024-03-29T11:14:18.539 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.560+0000 7f7d5cfef640 10 monclient(hunting): _negotiate 2024-03-29T11:14:18.539 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.560+0000 7f7d5cfef640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:18.539 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.560+0000 7f7d5cfef640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:18.539 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.560+0000 7f7d5cfef640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:14:18.539 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.560+0000 7f7d5cfef640 1 -- 172.21.15.157:0/3297738595 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d44003520 con 0x7f7d581457f0 2024-03-29T11:14:18.541 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.561+0000 7f7d5cfef640 1 -- 172.21.15.157:0/3297738595 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1218987744 0 0) 0x7f7d44003520 con 0x7f7d581457f0 2024-03-29T11:14:18.541 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.561+0000 7f7d5cfef640 1 -- 172.21.15.157:0/3297738595 >> v1:172.21.15.195:6789/0 conn(0x7f7d5813f650 legacy=0x7f7d580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:18.541 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.561+0000 7f7d5cfef640 1 -- 172.21.15.157:0/3297738595 >> v1:172.21.15.157:6789/0 conn(0x7f7d580a9d70 legacy=0x7f7d5814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:18.541 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 10 monclient: _finish_hunting 0 2024-03-29T11:14:18.541 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 1 monclient: found mon.noname-b 2024-03-29T11:14:18.541 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:18.541 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:14:18.541 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 1 -- 172.21.15.157:0/3297738595 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d58146340 con 0x7f7d581457f0 2024-03-29T11:14:18.541 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 10 monclient: _finish_auth 0 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 10 monclient: _check_auth_tickets 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 1 -- 172.21.15.157:0/3297738595 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7d4c001690 con 0x7f7d581457f0 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d5cfef640 10 monclient: dump: 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:18.542 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:18.543 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:18.543 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:18.543 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d60255640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:14:18.543 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.562+0000 7f7d60255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:18.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.564+0000 7f7d5cfef640 1 -- 172.21.15.157:0/3297738595 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d4c001920 con 0x7f7d581457f0 2024-03-29T11:14:18.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.564+0000 7f7d5cfef640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:18.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.564+0000 7f7d60255640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.564+0000 7f7d60255640 4 set_mon_vals no callback set 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.565+0000 7f7d5cfef640 1 -- 172.21.15.157:0/3297738595 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7d4c001cb0 con 0x7f7d581457f0 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.565+0000 7f7d5cfef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.565+0000 7f7d5cfef640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.565+0000 7f7d5cfef640 10 monclient: dump: 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:18.545 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:18.546 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:18.546 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:18.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.565+0000 7f7d60255640 10 monclient: shutdown 2024-03-29T11:14:18.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.565+0000 7f7d60255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:18.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.565+0000 7f7d60255640 1 -- 172.21.15.157:0/3297738595 >> v1:172.21.15.157:6790/0 conn(0x7f7d581457f0 legacy=0x7f7d5813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:18.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.565+0000 7f7d60255640 1 -- 172.21.15.157:0/3297738595 shutdown_connections 2024-03-29T11:14:18.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.566+0000 7f7d60255640 1 -- 172.21.15.157:0/3297738595 >> 172.21.15.157:0/3297738595 conn(0x7f7d58066a00 msgr2=0x7f7d58066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:18.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.566+0000 7f7d60255640 1 -- 172.21.15.157:0/3297738595 shutdown_connections 2024-03-29T11:14:18.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.566+0000 7f7d60255640 1 -- 172.21.15.157:0/3297738595 wait complete. 2024-03-29T11:14:18.550 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.567+0000 7f7d60255640 10 monclient: build_initial_monmap 2024-03-29T11:14:18.550 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.567+0000 7f7d60255640 10 monclient: monmap: 2024-03-29T11:14:18.550 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:18.550 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:18.550 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:14:18.550 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:14:18.550 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:18.550 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:18.550 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:18.550 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.568+0000 7f7d60255640 1 librados: starting msgr at 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.568+0000 7f7d60255640 1 librados: starting objecter 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.568+0000 7f7d60255640 1 Processor -- start 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.568+0000 7f7d60255640 1 -- start start 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.568+0000 7f7d60255640 1 librados: setting wanted keys 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.568+0000 7f7d60255640 1 librados: calling monclient init 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.568+0000 7f7d60255640 10 monclient: init 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.569+0000 7f7d60255640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.569+0000 7f7d60255640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.569+0000 7f7d60255640 10 monclient(hunting): picked mon.noname-a con 0x7f7d580a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.569+0000 7f7d60255640 10 monclient(hunting): picked mon.noname-c con 0x7f7d5813f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.569+0000 7f7d60255640 10 monclient(hunting): picked mon.noname-b con 0x7f7d581457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:18.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.569+0000 7f7d60255640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d580ab970 con 0x7f7d580a9d70 2024-03-29T11:14:18.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.569+0000 7f7d60255640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d580ac230 con 0x7f7d581457f0 2024-03-29T11:14:18.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.569+0000 7f7d60255640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d580ac510 con 0x7f7d5813f650 2024-03-29T11:14:18.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.569+0000 7f7d60255640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:18.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.569+0000 7f7d60255640 10 monclient(hunting): authenticate will time out at 1070.766204s 2024-03-29T11:14:18.557 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d5d7f0640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f7d5813f650 0x7f7d581428d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:52839/0 (socket says 172.21.15.157:52839) 2024-03-29T11:14:18.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d5d7f0640 1 -- 172.21.15.157:0/1170654793 learned_addr learned my addr 172.21.15.157:0/1170654793 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:18.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d4a7fc640 10 client.?.objecter ms_handle_connect 0x7f7d5813f650 2024-03-29T11:14:18.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:18.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167246771 0 0) 0x7f7d580ac510 con 0x7f7d5813f650 2024-03-29T11:14:18.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:18.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:18.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:18.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d4a7fc640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d3c004eb0 con 0x7f7d5813f650 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.573+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3337161033 0 0) 0x7f7d3c004eb0 con 0x7f7d5813f650 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.574+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d580ac510 con 0x7f7d5813f650 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.574+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7d54004080 con 0x7f7d5813f650 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.574+0000 7f7d4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.574+0000 7f7d4a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.574+0000 7f7d4a7fc640 10 monclient(hunting): dump: 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:18.559 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.574+0000 7f7d4a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.574+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4231127692 0 0) 0x7f7d580ac510 con 0x7f7d5813f650 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.574+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 >> v1:172.21.15.195:6789/0 conn(0x7f7d581457f0 legacy=0x7f7d581378d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 >> v1:172.21.15.157:6789/0 conn(0x7f7d580a9d70 legacy=0x7f7d580acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 1 monclient: found mon.c 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d580ac7f0 con 0x7f7d5813f650 2024-03-29T11:14:18.560 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 10 client.?.objecter ms_handle_connect 0x7f7d580a9d70 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d540043a0 con 0x7f7d5813f650 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d60255640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d60255640 10 monclient: _renew_subs 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d60255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d60255640 1 -- 172.21.15.157:0/1170654793 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d580ab600 con 0x7f7d5813f650 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4b7fe640 4 set_mon_vals no callback set 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d60255640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d60255640 10 monclient: _renew_subs 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d60255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:18.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d60255640 1 -- 172.21.15.157:0/1170654793 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d58144d70 con 0x7f7d5813f650 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d60255640 1 librados: init done 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7d54001320 con 0x7f7d5813f650 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.575+0000 7f7d4a7fc640 10 monclient: dump: 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:18.562 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.577+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f7d54001500 con 0x7f7d5813f650 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.577+0000 7f7d60255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.577+0000 7f7d60255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.577+0000 7f7d60255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.577+0000 7f7d60255640 1 -- 172.21.15.157:0/1170654793 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d58075f10 con 0x7f7d5813f650 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.578+0000 7f7d5dff1640 10 monclient: get_auth_request con 0x7f7d3c06dcd0 auth_method 0 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.579+0000 7f7d4a7fc640 10 client.4343.objecter ms_handle_connect 0x7f7d3c06dcd0 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.579+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f7d54084bf0 con 0x7f7d5813f650 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.579+0000 7f7d4a7fc640 10 client.4343.objecter ms_dispatch 0x7f7d58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.579+0000 7f7d4a7fc640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.579+0000 7f7d4a7fc640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:18.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.579+0000 7f7d4a7fc640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T11:14:18.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.587+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d540504d0 con 0x7f7d5813f650 2024-03-29T11:14:18.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.587+0000 7f7d4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:18.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.587+0000 7f7d4a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:18.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.913+0000 7f7d60255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:18.893 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.913+0000 7f7d60255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:18.893 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.913+0000 7f7d60255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:18.893 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.913+0000 7f7d60255640 1 -- 172.21.15.157:0/1170654793 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d580b3680 con 0x7f7d5813f650 2024-03-29T11:14:18.893 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:14:18.894 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:18.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.915+0000 7f7d4a7fc640 1 -- 172.21.15.157:0/1170654793 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f7d54052400 con 0x7f7d5813f650 2024-03-29T11:14:18.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.915+0000 7f7d4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:18.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.915+0000 7f7d4a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:18.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.919+0000 7f7d60255640 10 librados: watch_flush enter 2024-03-29T11:14:18.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.919+0000 7f7d60255640 10 librados: watch_flush exit 2024-03-29T11:14:18.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.919+0000 7f7d60255640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T11:14:18.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.919+0000 7f7d60255640 10 client.4343.objecter successfully canceled tick 2024-03-29T11:14:18.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.919+0000 7f7d60255640 1 -- 172.21.15.157:0/1170654793 >> v1:172.21.15.157:6816/1747058247 conn(0x7f7d3c06dcd0 legacy=0x7f7d3c070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:18.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.919+0000 7f7d60255640 10 monclient: shutdown 2024-03-29T11:14:18.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.919+0000 7f7d60255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:18.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.919+0000 7f7d60255640 1 -- 172.21.15.157:0/1170654793 >> v1:172.21.15.157:6790/0 conn(0x7f7d5813f650 legacy=0x7f7d581428d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:18.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.919+0000 7f7d60255640 1 -- 172.21.15.157:0/1170654793 shutdown_connections 2024-03-29T11:14:18.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.919+0000 7f7d60255640 1 -- 172.21.15.157:0/1170654793 >> 172.21.15.157:0/1170654793 conn(0x7f7d58066a00 msgr2=0x7f7d5813aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:18.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.920+0000 7f7d60255640 1 -- 172.21.15.157:0/1170654793 shutdown_connections 2024-03-29T11:14:18.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.921+0000 7f7d60255640 1 -- 172.21.15.157:0/1170654793 wait complete. 2024-03-29T11:14:18.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:18.921+0000 7f7d60255640 1 librados: shutdown 2024-03-29T11:14:18.916 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:25.724 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:25.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.895+0000 7fcdfb2e6640 10 monclient: get_monmap_and_config 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.895+0000 7fcdfb2e6640 10 monclient: build_initial_monmap 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.896+0000 7fcdfb2e6640 10 monclient: monmap: 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:14:25.897084+0000 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:14:25.897084+0000 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:14:25.874 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:25.875 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.897+0000 7fcdfb2e6640 1 Processor -- start 2024-03-29T11:14:25.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.898+0000 7fcdfb2e6640 1 -- start start 2024-03-29T11:14:25.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.898+0000 7fcdfb2e6640 10 monclient: init 2024-03-29T11:14:25.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.899+0000 7fcdfb2e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:25.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.899+0000 7fcdfb2e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:25.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.899+0000 7fcdfb2e6640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf41457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:25.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.899+0000 7fcdfb2e6640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf413f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:25.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.899+0000 7fcdfb2e6640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf40a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:25.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.899+0000 7fcdfb2e6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf40aaef0 con 0x7fcdf413f650 2024-03-29T11:14:25.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.899+0000 7fcdfb2e6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf4145bd0 con 0x7fcdf41457f0 2024-03-29T11:14:25.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.899+0000 7fcdfb2e6640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf4145eb0 con 0x7fcdf40a9d70 2024-03-29T11:14:25.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.899+0000 7fcdfb2e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:25.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.899+0000 7fcdfb2e6640 10 monclient(hunting): authenticate will time out at 1078.096059s 2024-03-29T11:14:25.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.903+0000 7fcdf9883640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fcdf40a9d70 0x7fcdf414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:53065/0 (socket says 172.21.15.157:53065) 2024-03-29T11:14:25.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.903+0000 7fcdf9883640 1 -- 172.21.15.157:0/4166094933 learned_addr learned my addr 172.21.15.157:0/4166094933 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:25.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.904+0000 7fcdebfff640 1 -- 172.21.15.157:0/4166094933 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218665606 0 0) 0x7fcdf4145eb0 con 0x7fcdf40a9d70 2024-03-29T11:14:25.883 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.904+0000 7fcdebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:25.883 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.904+0000 7fcdebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:25.883 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.904+0000 7fcdebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:25.883 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.904+0000 7fcdebfff640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:14:25.883 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.904+0000 7fcdebfff640 1 -- 172.21.15.157:0/4166094933 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde0003520 con 0x7fcdf40a9d70 2024-03-29T11:14:25.883 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.904+0000 7fcdebfff640 1 -- 172.21.15.157:0/4166094933 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2515374423 0 0) 0x7fcde0003520 con 0x7fcdf40a9d70 2024-03-29T11:14:25.883 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.904+0000 7fcdebfff640 1 -- 172.21.15.157:0/4166094933 >> v1:172.21.15.195:6789/0 conn(0x7fcdf41457f0 legacy=0x7fcdf413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.905+0000 7fcdebfff640 1 -- 172.21.15.157:0/4166094933 >> v1:172.21.15.157:6789/0 conn(0x7fcdf413f650 legacy=0x7fcdf40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 1 monclient: found mon.noname-b 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 1 -- 172.21.15.157:0/4166094933 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf4146340 con 0x7fcdf40a9d70 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 10 monclient: _finish_auth 0 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 1 -- 172.21.15.157:0/4166094933 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fcde4001690 con 0x7fcdf40a9d70 2024-03-29T11:14:25.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:25.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 10 monclient: dump: 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.906+0000 7fcdebfff640 1 -- 172.21.15.157:0/4166094933 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcde40019a0 con 0x7fcdf40a9d70 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.907+0000 7fcdebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:25.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.907+0000 7fcdebfff640 1 -- 172.21.15.157:0/4166094933 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fcde4001d30 con 0x7fcdf40a9d70 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.907+0000 7fcdebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.907+0000 7fcdebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.907+0000 7fcdebfff640 10 monclient: dump: 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:25.888 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:25.889 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:25.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.909+0000 7fcdfb2e6640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:14:25.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.909+0000 7fcdfb2e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:25.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.909+0000 7fcdfb2e6640 4 set_mon_vals no callback set 2024-03-29T11:14:25.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.910+0000 7fcdfb2e6640 10 monclient: shutdown 2024-03-29T11:14:25.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.910+0000 7fcdfb2e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:25.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.910+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/4166094933 >> v1:172.21.15.157:6790/0 conn(0x7fcdf40a9d70 legacy=0x7fcdf414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.910+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/4166094933 shutdown_connections 2024-03-29T11:14:25.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.910+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/4166094933 >> 172.21.15.157:0/4166094933 conn(0x7fcdf4066a00 msgr2=0x7fcdf4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:25.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.910+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/4166094933 shutdown_connections 2024-03-29T11:14:25.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.910+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/4166094933 wait complete. 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.911+0000 7fcdfb2e6640 10 monclient: build_initial_monmap 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.911+0000 7fcdfb2e6640 10 monclient: monmap: 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:14:25.893 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:25.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.912+0000 7fcdfb2e6640 1 librados: starting msgr at 2024-03-29T11:14:25.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.912+0000 7fcdfb2e6640 1 librados: starting objecter 2024-03-29T11:14:25.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.912+0000 7fcdfb2e6640 1 Processor -- start 2024-03-29T11:14:25.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.912+0000 7fcdfb2e6640 1 -- start start 2024-03-29T11:14:25.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.912+0000 7fcdfb2e6640 1 librados: setting wanted keys 2024-03-29T11:14:25.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.912+0000 7fcdfb2e6640 1 librados: calling monclient init 2024-03-29T11:14:25.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.912+0000 7fcdfb2e6640 10 monclient: init 2024-03-29T11:14:25.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.913+0000 7fcdfb2e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.913+0000 7fcdfb2e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.913+0000 7fcdfb2e6640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf40a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.913+0000 7fcdfb2e6640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf413f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcdf9883640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fcdf41457f0 0x7fcdf41378a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:53173/0 (socket says 172.21.15.157:53173) 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcdf9883640 1 -- 172.21.15.157:0/2361625977 learned_addr learned my addr 172.21.15.157:0/2361625977 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcdfb2e6640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf41457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcde9ffb640 10 client.?.objecter ms_handle_connect 0x7fcdf41457f0 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcde9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf4137fb0 con 0x7fcdf40a9d70 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf4142dc0 con 0x7fcdf413f650 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf406f280 con 0x7fcdf41457f0 2024-03-29T11:14:25.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcdfb2e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcdfb2e6640 10 monclient(hunting): authenticate will time out at 1078.113616s 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867047530 0 0) 0x7fcdf406f280 con 0x7fcdf41457f0 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcde9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcde9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcde9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcde9ffb640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.916+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd8004eb0 con 0x7fcdf41457f0 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.917+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2961226808 0 0) 0x7fcdd8004eb0 con 0x7fcdf41457f0 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.917+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdf406f280 con 0x7fcdf41457f0 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.917+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fcde4001650 con 0x7fcdf41457f0 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.917+0000 7fcde9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.917+0000 7fcde9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.917+0000 7fcde9ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:25.905 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:25.906 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:25.906 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:25.906 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:25.906 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:25.906 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:25.906 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:25.907 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:25.907 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:25.907 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:25.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.917+0000 7fcde9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:25.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.917+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3284925395 0 0) 0x7fcdf406f280 con 0x7fcdf41457f0 2024-03-29T11:14:25.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.917+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 >> v1:172.21.15.195:6789/0 conn(0x7fcdf413f650 legacy=0x7fcdf4137190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:25.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 >> v1:172.21.15.157:6789/0 conn(0x7fcdf40a9d70 legacy=0x7fcdf41426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 1 monclient: found mon.c 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf4141da0 con 0x7fcdf41457f0 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 10 client.?.objecter ms_handle_connect 0x7fcdf40a9d70 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:25.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcdfb2e6640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcdfb2e6640 10 monclient: _renew_subs 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcdfb2e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf413b270 con 0x7fcdf41457f0 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcde4004c70 con 0x7fcdf41457f0 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fcde4004f80 con 0x7fcdf41457f0 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcde9ffb640 10 monclient: dump: 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:25.909 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcdfb2e6640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcdfb2e6640 10 monclient: _renew_subs 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcdfb2e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf41db3b0 con 0x7fcdf41457f0 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcdfb2e6640 1 librados: init done 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.920+0000 7fcdeaffd640 4 set_mon_vals no callback set 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.923+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fcde401af50 con 0x7fcdf41457f0 2024-03-29T11:14:25.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.923+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fcde4084d80 con 0x7fcdf41457f0 2024-03-29T11:14:25.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.923+0000 7fcde9ffb640 10 client.4349.objecter ms_dispatch 0x7fcdf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:25.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.923+0000 7fcde9ffb640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:25.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.923+0000 7fcde9ffb640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:25.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.923+0000 7fcde9ffb640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T11:14:25.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.924+0000 7fcdf9082640 10 monclient: get_auth_request con 0x7fcdd806db20 auth_method 0 2024-03-29T11:14:25.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.924+0000 7fcde9ffb640 10 client.4349.objecter ms_handle_connect 0x7fcdd806db20 2024-03-29T11:14:25.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.924+0000 7fcdfb2e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:25.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.924+0000 7fcdfb2e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:25.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.924+0000 7fcdfb2e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:25.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.924+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf406f080 con 0x7fcdf41457f0 2024-03-29T11:14:25.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.932+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcde4050660 con 0x7fcdf41457f0 2024-03-29T11:14:25.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.932+0000 7fcde9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:25.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:25.932+0000 7fcde9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:26.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.251+0000 7fcdfb2e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:26.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.251+0000 7fcdfb2e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:26.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.251+0000 7fcdfb2e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:26.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.251+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdf4075f10 con 0x7fcdf41457f0 2024-03-29T11:14:26.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.253+0000 7fcde9ffb640 1 -- 172.21.15.157:0/2361625977 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fcde4052590 con 0x7fcdf41457f0 2024-03-29T11:14:26.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.253+0000 7fcde9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:26.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.253+0000 7fcde9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:26.232 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:14:26.232 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.257+0000 7fcdfb2e6640 10 librados: watch_flush enter 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.257+0000 7fcdfb2e6640 10 librados: watch_flush exit 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.257+0000 7fcdfb2e6640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.257+0000 7fcdfb2e6640 10 client.4349.objecter successfully canceled tick 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.257+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 >> v1:172.21.15.157:6816/1747058247 conn(0x7fcdd806db20 legacy=0x7fcdd806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.257+0000 7fcdfb2e6640 10 monclient: shutdown 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.258+0000 7fcdfb2e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.258+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 >> v1:172.21.15.157:6790/0 conn(0x7fcdf41457f0 legacy=0x7fcdf41378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.258+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 shutdown_connections 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.258+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 >> 172.21.15.157:0/2361625977 conn(0x7fcdf4066a00 msgr2=0x7fcdf40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:26.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.258+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 shutdown_connections 2024-03-29T11:14:26.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.258+0000 7fcdfb2e6640 1 -- 172.21.15.157:0/2361625977 wait complete. 2024-03-29T11:14:26.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:26.258+0000 7fcdfb2e6640 1 librados: shutdown 2024-03-29T11:14:26.253 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:33.061 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:33.223 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.245+0000 7f7add983640 10 monclient: get_monmap_and_config 2024-03-29T11:14:33.223 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.245+0000 7f7add983640 10 monclient: build_initial_monmap 2024-03-29T11:14:33.223 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.245+0000 7f7add983640 10 monclient: monmap: 2024-03-29T11:14:33.223 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:33.223 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:33.223 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:14:33.246395+0000 2024-03-29T11:14:33.223 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:14:33.246395+0000 2024-03-29T11:14:33.223 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:33.223 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:33.224 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:33.224 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:14:33.224 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:14:33.224 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:33.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.246+0000 7f7add983640 1 Processor -- start 2024-03-29T11:14:33.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.246+0000 7f7add983640 1 -- start start 2024-03-29T11:14:33.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.246+0000 7f7add983640 10 monclient: init 2024-03-29T11:14:33.226 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.247+0000 7f7add983640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:33.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.247+0000 7f7add983640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:33.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.247+0000 7f7add983640 10 monclient(hunting): picked mon.noname-a con 0x7f7ad81457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:33.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.247+0000 7f7add983640 10 monclient(hunting): picked mon.noname-c con 0x7f7ad813f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:33.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.247+0000 7f7add983640 10 monclient(hunting): picked mon.noname-b con 0x7f7ad80a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:33.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.247+0000 7f7add983640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad80aaef0 con 0x7f7ad81457f0 2024-03-29T11:14:33.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.247+0000 7f7add983640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad8145bd0 con 0x7f7ad813f650 2024-03-29T11:14:33.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.247+0000 7f7add983640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad8145eb0 con 0x7f7ad80a9d70 2024-03-29T11:14:33.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.247+0000 7f7add983640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:33.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.247+0000 7f7add983640 10 monclient(hunting): authenticate will time out at 1085.444260s 2024-03-29T11:14:33.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.250+0000 7f7ad6ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f7ad81457f0 0x7f7ad813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:33881/0 (socket says 172.21.15.157:33881) 2024-03-29T11:14:33.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.250+0000 7f7ad6ffd640 1 -- 172.21.15.157:0/3736206263 learned_addr learned my addr 172.21.15.157:0/3736206263 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:33.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533624866 0 0) 0x7f7ad80aaef0 con 0x7f7ad81457f0 2024-03-29T11:14:33.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:33.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:33.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:33.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:14:33.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ac0003520 con 0x7f7ad81457f0 2024-03-29T11:14:33.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56227344 0 0) 0x7f7ad8145eb0 con 0x7f7ad80a9d70 2024-03-29T11:14:33.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:33.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:33.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:33.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:14:33.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad80aaef0 con 0x7f7ad80a9d70 2024-03-29T11:14:33.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3222409389 0 0) 0x7f7ac0003520 con 0x7f7ad81457f0 2024-03-29T11:14:33.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 >> v1:172.21.15.157:6790/0 conn(0x7f7ad80a9d70 legacy=0x7f7ad814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:33.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 >> v1:172.21.15.195:6789/0 conn(0x7f7ad813f650 legacy=0x7f7ad80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:33.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:33.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:14:33.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:33.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:14:33.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad8146340 con 0x7f7ad81457f0 2024-03-29T11:14:33.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:33.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:33.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:33.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7ad0001a10 con 0x7f7ad81457f0 2024-03-29T11:14:33.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:33.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:33.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.254+0000 7f7ad5ffb640 10 monclient: dump: 2024-03-29T11:14:33.240 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:33.240 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:33.240 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:33.240 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:33.240 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:33.240 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:33.240 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:33.240 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:33.241 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:33.241 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:33.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.255+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ad0001b70 con 0x7f7ad81457f0 2024-03-29T11:14:33.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.255+0000 7f7ad5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:33.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.255+0000 7f7ad5ffb640 1 -- 172.21.15.157:0/3736206263 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7ad0001f80 con 0x7f7ad81457f0 2024-03-29T11:14:33.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.255+0000 7f7ad5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:33.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.255+0000 7f7ad5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:33.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.255+0000 7f7ad5ffb640 10 monclient: dump: 2024-03-29T11:14:33.242 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:33.242 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:33.243 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:33.243 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:33.243 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:33.243 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:33.243 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:33.243 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:33.243 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:33.243 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:33.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.255+0000 7f7add983640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:14:33.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.255+0000 7f7add983640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:33.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.255+0000 7f7add983640 4 set_mon_vals no callback set 2024-03-29T11:14:33.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.256+0000 7f7add983640 10 monclient: shutdown 2024-03-29T11:14:33.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.256+0000 7f7add983640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:33.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.256+0000 7f7add983640 1 -- 172.21.15.157:0/3736206263 >> v1:172.21.15.157:6789/0 conn(0x7f7ad81457f0 legacy=0x7f7ad813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:33.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.259+0000 7f7add983640 1 -- 172.21.15.157:0/3736206263 shutdown_connections 2024-03-29T11:14:33.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.259+0000 7f7add983640 1 -- 172.21.15.157:0/3736206263 >> 172.21.15.157:0/3736206263 conn(0x7f7ad8066a00 msgr2=0x7f7ad8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:33.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.259+0000 7f7add983640 1 -- 172.21.15.157:0/3736206263 shutdown_connections 2024-03-29T11:14:33.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.259+0000 7f7add983640 1 -- 172.21.15.157:0/3736206263 wait complete. 2024-03-29T11:14:33.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.260+0000 7f7add983640 10 monclient: build_initial_monmap 2024-03-29T11:14:33.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.260+0000 7f7add983640 10 monclient: monmap: 2024-03-29T11:14:33.246 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:33.246 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:33.246 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.260+0000 7f7add983640 1 librados: starting msgr at 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.260+0000 7f7add983640 1 librados: starting objecter 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.261+0000 7f7add983640 1 Processor -- start 2024-03-29T11:14:33.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.261+0000 7f7add983640 1 -- start start 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.261+0000 7f7add983640 1 librados: setting wanted keys 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.261+0000 7f7add983640 1 librados: calling monclient init 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.261+0000 7f7add983640 10 monclient: init 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.262+0000 7f7add983640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.262+0000 7f7add983640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.262+0000 7f7add983640 10 monclient(hunting): picked mon.noname-a con 0x7f7ad80a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.264+0000 7f7add983640 10 monclient(hunting): picked mon.noname-b con 0x7f7ad813f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.265+0000 7f7add983640 10 monclient(hunting): picked mon.noname-c con 0x7f7ad81457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.265+0000 7f7add983640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad806eb80 con 0x7f7ad80a9d70 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.265+0000 7f7add983640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad806ee60 con 0x7f7ad813f650 2024-03-29T11:14:33.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.265+0000 7f7add983640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad806f140 con 0x7f7ad81457f0 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.265+0000 7f7add983640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.265+0000 7f7add983640 10 monclient(hunting): authenticate will time out at 1085.461970s 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.265+0000 7f7ad67fc640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7ad813f650 0x7f7ad8142850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:46643/0 (socket says 172.21.15.157:46643) 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.265+0000 7f7ad67fc640 1 -- 172.21.15.157:0/110661757 learned_addr learned my addr 172.21.15.157:0/110661757 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.266+0000 7f7abffff640 10 client.?.objecter ms_handle_connect 0x7f7ad813f650 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.266+0000 7f7abffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.266+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537793307 0 0) 0x7f7ad806ee60 con 0x7f7ad813f650 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.266+0000 7f7abffff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.266+0000 7f7abffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.266+0000 7f7abffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.266+0000 7f7abffff640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.267+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ac4004eb0 con 0x7f7ad813f650 2024-03-29T11:14:33.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.269+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2522020118 0 0) 0x7f7ac4004eb0 con 0x7f7ad813f650 2024-03-29T11:14:33.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.269+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ad806ee60 con 0x7f7ad813f650 2024-03-29T11:14:33.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.270+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7acc001960 con 0x7f7ad813f650 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.270+0000 7f7abffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.270+0000 7f7abffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.270+0000 7f7abffff640 10 monclient(hunting): dump: 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:33.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.270+0000 7f7abffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:33.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.270+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 779233845 0 0) 0x7f7ad806ee60 con 0x7f7ad813f650 2024-03-29T11:14:33.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.270+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 >> v1:172.21.15.157:6790/0 conn(0x7f7ad81457f0 legacy=0x7f7ad81e5f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:33.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 >> v1:172.21.15.157:6789/0 conn(0x7f7ad80a9d70 legacy=0x7f7ad8141c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:33.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7abffff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:33.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7abffff640 1 monclient: found mon.b 2024-03-29T11:14:33.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7abffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:33.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7abffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:14:33.257 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad8137bb0 con 0x7f7ad813f650 2024-03-29T11:14:33.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7abffff640 10 monclient: _finish_auth 0 2024-03-29T11:14:33.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7abffff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:33.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7abffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:33.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7add983640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:14:33.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7add983640 10 monclient: _renew_subs 2024-03-29T11:14:33.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7add983640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:14:33.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7add983640 1 -- 172.21.15.157:0/110661757 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad806de90 con 0x7f7ad813f650 2024-03-29T11:14:33.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7add983640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:33.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7add983640 10 monclient: _renew_subs 2024-03-29T11:14:33.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7add983640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:14:33.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7add983640 1 -- 172.21.15.157:0/110661757 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad8144e10 con 0x7f7ad813f650 2024-03-29T11:14:33.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.271+0000 7f7add983640 1 librados: init done 2024-03-29T11:14:33.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.273+0000 7f7abffff640 10 client.4321.objecter ms_handle_connect 0x7f7ad80a9d70 2024-03-29T11:14:33.259 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.273+0000 7f7abffff640 10 client.4321.objecter resend_mon_ops 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.273+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7acc001d40 con 0x7f7ad813f650 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.273+0000 7f7abffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.273+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7acc001bc0 con 0x7f7ad813f650 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.273+0000 7f7abffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.273+0000 7f7abffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.273+0000 7f7abffff640 10 monclient: dump: 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:33.261 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:33.262 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:33.262 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:33.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.273+0000 7f7ad57fa640 4 set_mon_vals no callback set 2024-03-29T11:14:33.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.274+0000 7f7add983640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:33.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.274+0000 7f7add983640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:33.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.274+0000 7f7add983640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:14:33.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.274+0000 7f7add983640 1 -- 172.21.15.157:0/110661757 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab8005080 con 0x7f7ad813f650 2024-03-29T11:14:33.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.278+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f7acc003a40 con 0x7f7ad813f650 2024-03-29T11:14:33.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.279+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f7acc095a90 con 0x7f7ad813f650 2024-03-29T11:14:33.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.279+0000 7f7abffff640 10 client.4321.objecter ms_dispatch 0x7f7ad8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:33.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.279+0000 7f7abffff640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:33.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.279+0000 7f7abffff640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:33.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.279+0000 7f7abffff640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-29T11:14:33.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.279+0000 7f7ad6ffd640 10 monclient: get_auth_request con 0x7f7ac406dbd0 auth_method 0 2024-03-29T11:14:33.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.283+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7acc0611a0 con 0x7f7ad813f650 2024-03-29T11:14:33.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.283+0000 7f7abffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:33.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.283+0000 7f7abffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:33.264 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.284+0000 7f7abffff640 10 client.4321.objecter ms_handle_connect 0x7f7ac406dbd0 2024-03-29T11:14:33.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.599+0000 7f7add983640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:33.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.599+0000 7f7add983640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:33.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.599+0000 7f7add983640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:14:33.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.599+0000 7f7add983640 1 -- 172.21.15.157:0/110661757 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ab8005330 con 0x7f7ad813f650 2024-03-29T11:14:33.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.600+0000 7f7abffff640 1 -- 172.21.15.157:0/110661757 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f7acc0632a0 con 0x7f7ad813f650 2024-03-29T11:14:33.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.600+0000 7f7abffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:33.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.600+0000 7f7abffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:33.578 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:14:33.578 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:33.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.603+0000 7f7add983640 10 librados: watch_flush enter 2024-03-29T11:14:33.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.603+0000 7f7add983640 10 librados: watch_flush exit 2024-03-29T11:14:33.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.603+0000 7f7add983640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-29T11:14:33.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.603+0000 7f7add983640 10 client.4321.objecter successfully canceled tick 2024-03-29T11:14:33.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.604+0000 7f7add983640 1 -- 172.21.15.157:0/110661757 >> v1:172.21.15.157:6816/1747058247 conn(0x7f7ac406dbd0 legacy=0x7f7ac4070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:33.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.604+0000 7f7add983640 10 monclient: shutdown 2024-03-29T11:14:33.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.604+0000 7f7add983640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:33.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.604+0000 7f7add983640 1 -- 172.21.15.157:0/110661757 >> v1:172.21.15.195:6789/0 conn(0x7f7ad813f650 legacy=0x7f7ad8142850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:33.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.607+0000 7f7add983640 1 -- 172.21.15.157:0/110661757 shutdown_connections 2024-03-29T11:14:33.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.607+0000 7f7add983640 1 -- 172.21.15.157:0/110661757 >> 172.21.15.157:0/110661757 conn(0x7f7ad8066a00 msgr2=0x7f7ad8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:33.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.611+0000 7f7add983640 1 -- 172.21.15.157:0/110661757 shutdown_connections 2024-03-29T11:14:33.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.611+0000 7f7add983640 1 -- 172.21.15.157:0/110661757 wait complete. 2024-03-29T11:14:33.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:33.612+0000 7f7add983640 1 librados: shutdown 2024-03-29T11:14:33.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:40.424 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:40.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.588+0000 7f1daa240640 10 monclient: get_monmap_and_config 2024-03-29T11:14:40.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.588+0000 7f1daa240640 10 monclient: build_initial_monmap 2024-03-29T11:14:40.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.588+0000 7f1daa240640 10 monclient: monmap: 2024-03-29T11:14:40.570 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:40.571 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:40.571 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:14:40.589851+0000 2024-03-29T11:14:40.571 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:14:40.589851+0000 2024-03-29T11:14:40.571 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:40.571 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:40.571 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:40.571 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:14:40.571 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:14:40.571 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:40.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.592+0000 7f1daa240640 1 Processor -- start 2024-03-29T11:14:40.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.592+0000 7f1daa240640 1 -- start start 2024-03-29T11:14:40.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.592+0000 7f1daa240640 10 monclient: init 2024-03-29T11:14:40.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.593+0000 7f1daa240640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.593+0000 7f1daa240640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.593+0000 7f1daa240640 10 monclient(hunting): picked mon.noname-c con 0x7f1da40b1f60 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.593+0000 7f1daa240640 10 monclient(hunting): picked mon.noname-b con 0x7f1da4071eb0 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.593+0000 7f1daa240640 10 monclient(hunting): picked mon.noname-a con 0x7f1da4143d10 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.593+0000 7f1daa240640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da406ef80 con 0x7f1da4143d10 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.593+0000 7f1daa240640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da406f260 con 0x7f1da40b1f60 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.593+0000 7f1daa240640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da40b40c0 con 0x7f1da4071eb0 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.593+0000 7f1daa240640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.593+0000 7f1daa240640 10 monclient(hunting): authenticate will time out at 1092.790170s 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.595+0000 7f1da3fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1da4143d10 0x7f1da4146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:59183/0 (socket says 172.21.15.157:59183) 2024-03-29T11:14:40.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.595+0000 7f1da3fff640 1 -- 172.21.15.157:0/3554266648 learned_addr learned my addr 172.21.15.157:0/3554266648 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:40.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.596+0000 7f1da27fc640 1 -- 172.21.15.157:0/3554266648 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043491446 0 0) 0x7f1da406ef80 con 0x7f1da4143d10 2024-03-29T11:14:40.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.596+0000 7f1da27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:40.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.596+0000 7f1da27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:40.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.596+0000 7f1da27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:40.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.596+0000 7f1da27fc640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:14:40.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.596+0000 7f1da27fc640 1 -- 172.21.15.157:0/3554266648 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c003520 con 0x7f1da4143d10 2024-03-29T11:14:40.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 1 -- 172.21.15.157:0/3554266648 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 15743137 0 0) 0x7f1d8c003520 con 0x7f1da4143d10 2024-03-29T11:14:40.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 1 -- 172.21.15.157:0/3554266648 >> v1:172.21.15.157:6790/0 conn(0x7f1da4071eb0 legacy=0x7f1da4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:40.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 1 -- 172.21.15.157:0/3554266648 >> v1:172.21.15.195:6789/0 conn(0x7f1da40b1f60 legacy=0x7f1da4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:40.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:40.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 1 monclient: found mon.noname-a 2024-03-29T11:14:40.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:40.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:14:40.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 1 -- 172.21.15.157:0/3554266648 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da4155d70 con 0x7f1da4143d10 2024-03-29T11:14:40.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1daa240640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1daa240640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 1 -- 172.21.15.157:0/3554266648 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1d98001690 con 0x7f1da4143d10 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.600+0000 7f1da27fc640 10 monclient: dump: 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:40.582 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.601+0000 7f1daa240640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.601+0000 7f1da27fc640 1 -- 172.21.15.157:0/3554266648 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d98001920 con 0x7f1da4143d10 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.601+0000 7f1da27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.601+0000 7f1da27fc640 1 -- 172.21.15.157:0/3554266648 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1d98001cb0 con 0x7f1da4143d10 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.601+0000 7f1daa240640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.601+0000 7f1daa240640 4 set_mon_vals no callback set 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.601+0000 7f1da27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.601+0000 7f1da27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.601+0000 7f1da27fc640 10 monclient: dump: 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:40.583 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.602+0000 7f1daa240640 10 monclient: shutdown 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.602+0000 7f1daa240640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.602+0000 7f1daa240640 1 -- 172.21.15.157:0/3554266648 >> v1:172.21.15.157:6789/0 conn(0x7f1da4143d10 legacy=0x7f1da4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.602+0000 7f1daa240640 1 -- 172.21.15.157:0/3554266648 shutdown_connections 2024-03-29T11:14:40.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.602+0000 7f1daa240640 1 -- 172.21.15.157:0/3554266648 >> 172.21.15.157:0/3554266648 conn(0x7f1da4066a00 msgr2=0x7f1da4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:40.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.603+0000 7f1daa240640 1 -- 172.21.15.157:0/3554266648 shutdown_connections 2024-03-29T11:14:40.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.603+0000 7f1daa240640 1 -- 172.21.15.157:0/3554266648 wait complete. 2024-03-29T11:14:40.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.603+0000 7f1daa240640 10 monclient: build_initial_monmap 2024-03-29T11:14:40.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.604+0000 7f1daa240640 10 monclient: monmap: 2024-03-29T11:14:40.589 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:40.589 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:40.589 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:14:40.589 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:14:40.589 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:40.589 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.604+0000 7f1daa240640 1 librados: starting msgr at 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.604+0000 7f1daa240640 1 librados: starting objecter 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.604+0000 7f1daa240640 1 Processor -- start 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.604+0000 7f1daa240640 1 -- start start 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.604+0000 7f1daa240640 1 librados: setting wanted keys 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.604+0000 7f1daa240640 1 librados: calling monclient init 2024-03-29T11:14:40.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.604+0000 7f1daa240640 10 monclient: init 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.605+0000 7f1daa240640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.605+0000 7f1daa240640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.605+0000 7f1daa240640 10 monclient(hunting): picked mon.noname-b con 0x7f1da40b1f60 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.608+0000 7f1da2ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f1da4143d10 0x7f1da4070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:59325/0 (socket says 172.21.15.157:59325) 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.608+0000 7f1da2ffd640 1 -- 172.21.15.157:0/3568767535 learned_addr learned my addr 172.21.15.157:0/3568767535 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.608+0000 7f1daa240640 10 monclient(hunting): picked mon.noname-a con 0x7f1da4143d10 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.608+0000 7f1daa240640 10 monclient(hunting): picked mon.noname-c con 0x7f1da4130810 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da40714f0 con 0x7f1da4143d10 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da4154510 con 0x7f1da40b1f60 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da41547f0 con 0x7f1da4130810 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1daa240640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1daa240640 10 monclient(hunting): authenticate will time out at 1092.805965s 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 10 client.?.objecter ms_handle_connect 0x7f1da4143d10 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:40.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653028841 0 0) 0x7f1da40714f0 con 0x7f1da4143d10 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d90004eb0 con 0x7f1da4143d10 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1485909920 0 0) 0x7f1d90004eb0 con 0x7f1da4143d10 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1da40714f0 con 0x7f1da4143d10 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1d9c001ce0 con 0x7f1da4143d10 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 10 monclient(hunting): dump: 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:40.592 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:40.593 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:40.593 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:40.593 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:40.593 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:40.593 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:40.593 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:40.593 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:40.593 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:40.593 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3900899994 0 0) 0x7f1da40714f0 con 0x7f1da4143d10 2024-03-29T11:14:40.593 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.609+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 >> v1:172.21.15.157:6790/0 conn(0x7f1da4130810 legacy=0x7f1da41eed20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:40.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.612+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 >> v1:172.21.15.195:6789/0 conn(0x7f1da40b1f60 legacy=0x7f1da4141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:40.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1d8bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:40.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1d8bfff640 1 monclient: found mon.a 2024-03-29T11:14:40.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1d8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:40.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1d8bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:14:40.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da4154ad0 con 0x7f1da4143d10 2024-03-29T11:14:40.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1d8bfff640 10 monclient: _finish_auth 0 2024-03-29T11:14:40.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1d8bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:40.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1d8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1daa240640 5 monclient: authenticate success, global_id 4347 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1daa240640 10 monclient: _renew_subs 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1daa240640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da4154db0 con 0x7f1da4143d10 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1daa240640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1daa240640 10 monclient: _renew_subs 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1daa240640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da41f0440 con 0x7f1da4143d10 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.613+0000 7f1daa240640 1 librados: init done 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1daa240640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1daa240640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1daa240640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da4130bf0 con 0x7f1da4143d10 2024-03-29T11:14:40.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d9c001e40 con 0x7f1da4143d10 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1d8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1d9c003dc0 con 0x7f1da4143d10 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1d8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1d8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1d8bfff640 10 monclient: dump: 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:40.597 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:40.598 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.616+0000 7f1da17fa640 4 set_mon_vals no callback set 2024-03-29T11:14:40.598 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.619+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f1d9c003fe0 con 0x7f1da4143d10 2024-03-29T11:14:40.598 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.620+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f1d9c084e10 con 0x7f1da4143d10 2024-03-29T11:14:40.598 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.620+0000 7f1d8bfff640 10 client.4347.objecter ms_dispatch 0x7f1da4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:40.598 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.620+0000 7f1d8bfff640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:40.598 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.620+0000 7f1d8bfff640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:40.598 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.620+0000 7f1d8bfff640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T11:14:40.599 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.621+0000 7f1da37fe640 10 monclient: get_auth_request con 0x7f1d9006dc90 auth_method 0 2024-03-29T11:14:40.599 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.621+0000 7f1d8bfff640 10 client.4347.objecter ms_handle_connect 0x7f1d9006dc90 2024-03-29T11:14:40.608 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.629+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1da4130bf0 con 0x7f1da4143d10 2024-03-29T11:14:40.609 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.629+0000 7f1d8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:40.609 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.629+0000 7f1d8bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:40.924 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.946+0000 7f1daa240640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:40.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.946+0000 7f1daa240640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:40.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.947+0000 7f1daa240640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:14:40.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.947+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1da4130bf0 con 0x7f1da4143d10 2024-03-29T11:14:40.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.948+0000 7f1d8bfff640 1 -- 172.21.15.157:0/3568767535 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f1da4130bf0 con 0x7f1da4143d10 2024-03-29T11:14:40.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.948+0000 7f1d8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:40.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.948+0000 7f1d8bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:40.926 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:14:40.926 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:40.930 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.952+0000 7f1daa240640 10 librados: watch_flush enter 2024-03-29T11:14:40.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.954+0000 7f1daa240640 10 librados: watch_flush exit 2024-03-29T11:14:40.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.954+0000 7f1daa240640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T11:14:40.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.954+0000 7f1daa240640 10 client.4347.objecter successfully canceled tick 2024-03-29T11:14:40.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.954+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 >> v1:172.21.15.157:6816/1747058247 conn(0x7f1d9006dc90 legacy=0x7f1d90070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:40.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.954+0000 7f1daa240640 10 monclient: shutdown 2024-03-29T11:14:40.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.954+0000 7f1daa240640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:40.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.954+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 >> v1:172.21.15.157:6789/0 conn(0x7f1da4143d10 legacy=0x7f1da4070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:40.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.955+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 shutdown_connections 2024-03-29T11:14:40.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.955+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 >> 172.21.15.157:0/3568767535 conn(0x7f1da4066a00 msgr2=0x7f1da4146960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:40.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.956+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 shutdown_connections 2024-03-29T11:14:40.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.959+0000 7f1daa240640 1 -- 172.21.15.157:0/3568767535 wait complete. 2024-03-29T11:14:40.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:40.959+0000 7f1daa240640 1 librados: shutdown 2024-03-29T11:14:40.958 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:47.765 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:47.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.934+0000 7f842a53a640 10 monclient: get_monmap_and_config 2024-03-29T11:14:47.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.934+0000 7f842a53a640 10 monclient: build_initial_monmap 2024-03-29T11:14:47.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.934+0000 7f842a53a640 10 monclient: monmap: 2024-03-29T11:14:47.912 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:47.912 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:47.912 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:14:47.935686+0000 2024-03-29T11:14:47.913 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:14:47.935686+0000 2024-03-29T11:14:47.913 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:47.913 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:47.913 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:47.913 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:14:47.913 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:14:47.913 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:47.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.935+0000 7f842a53a640 1 Processor -- start 2024-03-29T11:14:47.914 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.935+0000 7f842a53a640 1 -- start start 2024-03-29T11:14:47.914 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.935+0000 7f842a53a640 10 monclient: init 2024-03-29T11:14:47.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.936+0000 7f842a53a640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:47.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.936+0000 7f842a53a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:47.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.936+0000 7f842a53a640 10 monclient(hunting): picked mon.noname-b con 0x7f84241457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:47.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.936+0000 7f842a53a640 10 monclient(hunting): picked mon.noname-a con 0x7f842413f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:47.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.936+0000 7f842a53a640 10 monclient(hunting): picked mon.noname-c con 0x7f84240a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:47.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.936+0000 7f842a53a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84240aaef0 con 0x7f842413f650 2024-03-29T11:14:47.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.936+0000 7f842a53a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8424145bd0 con 0x7f84240a9d70 2024-03-29T11:14:47.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.936+0000 7f842a53a640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8424145eb0 con 0x7f84241457f0 2024-03-29T11:14:47.916 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.936+0000 7f842a53a640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:47.916 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.936+0000 7f842a53a640 10 monclient(hunting): authenticate will time out at 1100.133766s 2024-03-29T11:14:47.916 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.937+0000 7f8423fff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f84241457f0 0x7f842413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:36472/0 (socket says 172.21.15.157:36472) 2024-03-29T11:14:47.916 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.937+0000 7f8423fff640 1 -- 172.21.15.157:0/1166342837 learned_addr learned my addr 172.21.15.157:0/1166342837 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:47.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749572920 0 0) 0x7f8424145bd0 con 0x7f84240a9d70 2024-03-29T11:14:47.918 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.918 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.918 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.918 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:14:47.918 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8410003520 con 0x7f84240a9d70 2024-03-29T11:14:47.918 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340462768 0 0) 0x7f8424145eb0 con 0x7f84241457f0 2024-03-29T11:14:47.918 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.918 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.918 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8424145bd0 con 0x7f84241457f0 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1121864107 0 0) 0x7f8410003520 con 0x7f84240a9d70 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 >> v1:172.21.15.157:6790/0 conn(0x7f84241457f0 legacy=0x7f842413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 >> v1:172.21.15.157:6789/0 conn(0x7f842413f650 legacy=0x7f84240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 1 monclient: found mon.noname-c 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8424146340 con 0x7f84240a9d70 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:47.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f840c001690 con 0x7f84240a9d70 2024-03-29T11:14:47.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f842a53a640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:14:47.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f842a53a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:47.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:47.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f8422ffd640 10 monclient: dump: 2024-03-29T11:14:47.920 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:47.920 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:47.920 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.920 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.938+0000 7f842a53a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.939+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f840c0017f0 con 0x7f84240a9d70 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.939+0000 7f8422ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.939+0000 7f8422ffd640 1 -- 172.21.15.157:0/1166342837 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f840c001c40 con 0x7f84240a9d70 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.939+0000 7f8422ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.939+0000 7f8422ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.939+0000 7f8422ffd640 10 monclient: dump: 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.939+0000 7f842a53a640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:47.922 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.939+0000 7f842a53a640 4 set_mon_vals no callback set 2024-03-29T11:14:47.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.940+0000 7f842a53a640 10 monclient: shutdown 2024-03-29T11:14:47.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.940+0000 7f842a53a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:47.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.940+0000 7f842a53a640 1 -- 172.21.15.157:0/1166342837 >> v1:172.21.15.195:6789/0 conn(0x7f84240a9d70 legacy=0x7f842414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.940+0000 7f842a53a640 1 -- 172.21.15.157:0/1166342837 shutdown_connections 2024-03-29T11:14:47.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.940+0000 7f842a53a640 1 -- 172.21.15.157:0/1166342837 >> 172.21.15.157:0/1166342837 conn(0x7f8424066a00 msgr2=0x7f8424066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:47.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.940+0000 7f842a53a640 1 -- 172.21.15.157:0/1166342837 shutdown_connections 2024-03-29T11:14:47.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.940+0000 7f842a53a640 1 -- 172.21.15.157:0/1166342837 wait complete. 2024-03-29T11:14:47.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.941+0000 7f842a53a640 10 monclient: build_initial_monmap 2024-03-29T11:14:47.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.941+0000 7f842a53a640 10 monclient: monmap: 2024-03-29T11:14:47.933 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:47.933 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:47.933 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.941+0000 7f842a53a640 1 librados: starting msgr at 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.941+0000 7f842a53a640 1 librados: starting objecter 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.941+0000 7f842a53a640 1 Processor -- start 2024-03-29T11:14:47.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.942+0000 7f842a53a640 1 -- start start 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.942+0000 7f842a53a640 1 librados: setting wanted keys 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.942+0000 7f842a53a640 1 librados: calling monclient init 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.942+0000 7f842a53a640 10 monclient: init 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f842a53a640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f842a53a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f842a53a640 10 monclient(hunting): picked mon.noname-a con 0x7f84240a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f842a53a640 10 monclient(hunting): picked mon.noname-b con 0x7f842413f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f842a53a640 10 monclient(hunting): picked mon.noname-c con 0x7f84241457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f842a53a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842406f0d0 con 0x7f84240a9d70 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f842a53a640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8424138ee0 con 0x7f842413f650 2024-03-29T11:14:47.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f8428ad7640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f84241457f0 0x7f84241e5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:36794/0 (socket says 172.21.15.157:36794) 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f8428ad7640 1 -- 172.21.15.157:0/780025148 learned_addr learned my addr 172.21.15.157:0/780025148 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842406fa50 con 0x7f84241457f0 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f842a53a640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f842a53a640 10 monclient(hunting): authenticate will time out at 1100.140493s 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f8420ff9640 10 client.?.objecter ms_handle_connect 0x7f84241457f0 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f8420ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f8420ff9640 10 client.?.objecter ms_handle_connect 0x7f84240a9d70 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f8420ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.943+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812026837 0 0) 0x7f842406fa50 con 0x7f84241457f0 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:14:47.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8404004eb0 con 0x7f84241457f0 2024-03-29T11:14:47.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 client.?.objecter ms_handle_connect 0x7f842413f650 2024-03-29T11:14:47.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:47.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749942234 0 0) 0x7f842406f0d0 con 0x7f84240a9d70 2024-03-29T11:14:47.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:14:47.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f842406fa50 con 0x7f84240a9d70 2024-03-29T11:14:47.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952217485 0 0) 0x7f8424138ee0 con 0x7f842413f650 2024-03-29T11:14:47.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:14:47.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:47.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:47.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:14:47.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f842406f0d0 con 0x7f842413f650 2024-03-29T11:14:47.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1694042418 0 0) 0x7f8404004eb0 con 0x7f84241457f0 2024-03-29T11:14:47.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8424138ee0 con 0x7f84241457f0 2024-03-29T11:14:47.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f840c001b40 con 0x7f84241457f0 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): dump: 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:47.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.944+0000 7f8420ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4290198060 0 0) 0x7f842406fa50 con 0x7f84240a9d70 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8404004eb0 con 0x7f84240a9d70 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3018341705 0 0) 0x7f842406f0d0 con 0x7f842413f650 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f842406fa50 con 0x7f842413f650 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f841c001a30 con 0x7f84240a9d70 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 10 monclient(hunting): dump: 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:47.940 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f84140017f0 con 0x7f842413f650 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 10 monclient(hunting): dump: 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.941 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2109771099 0 0) 0x7f8424138ee0 con 0x7f84241457f0 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 >> v1:172.21.15.195:6789/0 conn(0x7f842413f650 legacy=0x7f8424071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.945+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 >> v1:172.21.15.157:6789/0 conn(0x7f84240a9d70 legacy=0x7f8424070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f8420ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f8420ff9640 1 monclient: found mon.c 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f8420ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:47.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f8420ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842406fd30 con 0x7f84241457f0 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f8420ff9640 10 monclient: _finish_auth 0 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f8420ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f8420ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f842a53a640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f842a53a640 10 monclient: _renew_subs 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f842a53a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8424070010 con 0x7f84241457f0 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f842a53a640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f842a53a640 10 monclient: _renew_subs 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f842a53a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8424140390 con 0x7f84241457f0 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.946+0000 7f842a53a640 1 librados: init done 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.947+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f840c0038a0 con 0x7f84241457f0 2024-03-29T11:14:47.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.947+0000 7f8420ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:47.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.948+0000 7f842a53a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:47.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.948+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f840c003c70 con 0x7f84241457f0 2024-03-29T11:14:47.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.948+0000 7f8420ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:47.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.948+0000 7f8420ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:47.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.948+0000 7f84227fc640 4 set_mon_vals no callback set 2024-03-29T11:14:47.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.948+0000 7f8420ff9640 10 monclient: dump: 2024-03-29T11:14:47.944 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:47.944 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:47.944 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.944 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.948+0000 7f842a53a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.949+0000 7f842a53a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.949+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8424075f10 con 0x7f84241457f0 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.950+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f840c003dd0 con 0x7f84241457f0 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.950+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f840c084b00 con 0x7f84241457f0 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.950+0000 7f8420ff9640 10 client.4358.objecter ms_dispatch 0x7f8424000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.950+0000 7f8420ff9640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.950+0000 7f8420ff9640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:47.945 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.950+0000 7f8420ff9640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-29T11:14:47.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.951+0000 7f8423fff640 10 monclient: get_auth_request con 0x7f840406e2b0 auth_method 0 2024-03-29T11:14:47.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.951+0000 7f8420ff9640 10 client.4358.objecter ms_handle_connect 0x7f840406e2b0 2024-03-29T11:14:47.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.957+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f840c0503e0 con 0x7f84241457f0 2024-03-29T11:14:47.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.957+0000 7f8420ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:47.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:47.957+0000 7f8420ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:48.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.393+0000 7f842a53a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:48.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.393+0000 7f842a53a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:48.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.393+0000 7f842a53a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:14:48.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.393+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84240b3680 con 0x7f84241457f0 2024-03-29T11:14:48.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.395+0000 7f8420ff9640 1 -- 172.21.15.157:0/780025148 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f840c052310 con 0x7f84241457f0 2024-03-29T11:14:48.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.395+0000 7f8420ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:48.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.395+0000 7f8420ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:48.373 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:14:48.374 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:48.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.400+0000 7f842a53a640 10 librados: watch_flush enter 2024-03-29T11:14:48.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.400+0000 7f842a53a640 10 librados: watch_flush exit 2024-03-29T11:14:48.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.400+0000 7f842a53a640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-29T11:14:48.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.400+0000 7f842a53a640 10 client.4358.objecter successfully canceled tick 2024-03-29T11:14:48.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.400+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 >> v1:172.21.15.157:6816/1747058247 conn(0x7f840406e2b0 legacy=0x7f8404070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:48.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.400+0000 7f842a53a640 10 monclient: shutdown 2024-03-29T11:14:48.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.400+0000 7f842a53a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:48.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.400+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 >> v1:172.21.15.157:6790/0 conn(0x7f84241457f0 legacy=0x7f84241e5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:48.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.401+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 shutdown_connections 2024-03-29T11:14:48.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.401+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 >> 172.21.15.157:0/780025148 conn(0x7f8424066a00 msgr2=0x7f84240b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:48.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.401+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 shutdown_connections 2024-03-29T11:14:48.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.401+0000 7f842a53a640 1 -- 172.21.15.157:0/780025148 wait complete. 2024-03-29T11:14:48.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:48.402+0000 7f842a53a640 1 librados: shutdown 2024-03-29T11:14:48.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:55.202 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:55.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.374+0000 7f9b6973d640 10 monclient: get_monmap_and_config 2024-03-29T11:14:55.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.374+0000 7f9b6973d640 10 monclient: build_initial_monmap 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.374+0000 7f9b6973d640 10 monclient: monmap: 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:14:55.375165+0000 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:14:55.375165+0000 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:14:55.352 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:55.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.375+0000 7f9b6973d640 1 Processor -- start 2024-03-29T11:14:55.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.375+0000 7f9b6973d640 1 -- start start 2024-03-29T11:14:55.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.375+0000 7f9b6973d640 10 monclient: init 2024-03-29T11:14:55.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.376+0000 7f9b6973d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:55.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.376+0000 7f9b6973d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:55.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.376+0000 7f9b6973d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b641457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:55.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.376+0000 7f9b6973d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b6413f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b6973d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b640a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b6973d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b640aaef0 con 0x7f9b640a9d70 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b62ffd640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f9b641457f0 0x7f9b6413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:54730/0 (socket says 172.21.15.157:54730) 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b62ffd640 1 -- 172.21.15.157:0/3074802004 learned_addr learned my addr 172.21.15.157:0/3074802004 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b6973d640 1 -- 172.21.15.157:0/3074802004 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b64145bd0 con 0x7f9b6413f650 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b6973d640 1 -- 172.21.15.157:0/3074802004 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b64145eb0 con 0x7f9b641457f0 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b6973d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b6973d640 10 monclient(hunting): authenticate will time out at 1107.574485s 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b61ffb640 1 -- 172.21.15.157:0/3074802004 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836370384 0 0) 0x7f9b64145eb0 con 0x7f9b641457f0 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b61ffb640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:14:55.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.377+0000 7f9b61ffb640 1 -- 172.21.15.157:0/3074802004 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b4c003520 con 0x7f9b641457f0 2024-03-29T11:14:55.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.378+0000 7f9b61ffb640 1 -- 172.21.15.157:0/3074802004 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986408005 0 0) 0x7f9b4c003520 con 0x7f9b641457f0 2024-03-29T11:14:55.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.379+0000 7f9b61ffb640 1 -- 172.21.15.157:0/3074802004 >> v1:172.21.15.195:6789/0 conn(0x7f9b6413f650 legacy=0x7f9b640b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:55.359 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.381+0000 7f9b61ffb640 1 -- 172.21.15.157:0/3074802004 >> v1:172.21.15.157:6789/0 conn(0x7f9b640a9d70 legacy=0x7f9b6414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 1 monclient: found mon.noname-b 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 1 -- 172.21.15.157:0/3074802004 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b64146340 con 0x7f9b641457f0 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 1 -- 172.21.15.157:0/3074802004 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9b54001690 con 0x7f9b641457f0 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:55.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.383+0000 7f9b61ffb640 10 monclient: dump: 2024-03-29T11:14:55.362 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:55.362 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:55.362 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:55.362 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:55.362 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:55.362 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:55.362 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:55.362 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:55.362 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:55.362 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:55.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.385+0000 7f9b6973d640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:14:55.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.385+0000 7f9b6973d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:55.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.386+0000 7f9b61ffb640 1 -- 172.21.15.157:0/3074802004 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b54001860 con 0x7f9b641457f0 2024-03-29T11:14:55.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.386+0000 7f9b61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:55.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.386+0000 7f9b6973d640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:55.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.386+0000 7f9b6973d640 4 set_mon_vals no callback set 2024-03-29T11:14:55.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.387+0000 7f9b61ffb640 1 -- 172.21.15.157:0/3074802004 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9b54001c70 con 0x7f9b641457f0 2024-03-29T11:14:55.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.387+0000 7f9b61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:55.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.387+0000 7f9b61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:55.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.387+0000 7f9b61ffb640 10 monclient: dump: 2024-03-29T11:14:55.365 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:55.365 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:55.365 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:55.365 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:55.366 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:55.366 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:55.366 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:55.366 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:55.366 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:55.366 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:55.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.387+0000 7f9b6973d640 10 monclient: shutdown 2024-03-29T11:14:55.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.387+0000 7f9b6973d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:55.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.387+0000 7f9b6973d640 1 -- 172.21.15.157:0/3074802004 >> v1:172.21.15.157:6790/0 conn(0x7f9b641457f0 legacy=0x7f9b6413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:55.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.387+0000 7f9b6973d640 1 -- 172.21.15.157:0/3074802004 shutdown_connections 2024-03-29T11:14:55.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.387+0000 7f9b6973d640 1 -- 172.21.15.157:0/3074802004 >> 172.21.15.157:0/3074802004 conn(0x7f9b64066a00 msgr2=0x7f9b64066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:55.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.388+0000 7f9b6973d640 1 -- 172.21.15.157:0/3074802004 shutdown_connections 2024-03-29T11:14:55.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.388+0000 7f9b6973d640 1 -- 172.21.15.157:0/3074802004 wait complete. 2024-03-29T11:14:55.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.389+0000 7f9b6973d640 10 monclient: build_initial_monmap 2024-03-29T11:14:55.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.389+0000 7f9b6973d640 10 monclient: monmap: 2024-03-29T11:14:55.367 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:14:55.367 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:55.367 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:14:55.368 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:14:55.368 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:55.368 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:55.368 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:14:55.368 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:14:55.368 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:14:55.368 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:55.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.390+0000 7f9b6973d640 1 librados: starting msgr at 2024-03-29T11:14:55.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.390+0000 7f9b6973d640 1 librados: starting objecter 2024-03-29T11:14:55.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.391+0000 7f9b6973d640 1 Processor -- start 2024-03-29T11:14:55.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.391+0000 7f9b6973d640 1 -- start start 2024-03-29T11:14:55.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.391+0000 7f9b6973d640 1 librados: setting wanted keys 2024-03-29T11:14:55.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.391+0000 7f9b6973d640 1 librados: calling monclient init 2024-03-29T11:14:55.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.391+0000 7f9b6973d640 10 monclient: init 2024-03-29T11:14:55.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.392+0000 7f9b6973d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:55.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.392+0000 7f9b6973d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:55.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.392+0000 7f9b6973d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b640a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:14:55.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.392+0000 7f9b6973d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b6413f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:14:55.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.392+0000 7f9b6973d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b641457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:14:55.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.392+0000 7f9b6973d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b641387a0 con 0x7f9b641457f0 2024-03-29T11:14:55.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.392+0000 7f9b6973d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b641d70d0 con 0x7f9b6413f650 2024-03-29T11:14:55.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.392+0000 7f9b6973d640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b640ace00 con 0x7f9b640a9d70 2024-03-29T11:14:55.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.392+0000 7f9b6973d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:55.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.392+0000 7f9b6973d640 10 monclient(hunting): authenticate will time out at 1107.589269s 2024-03-29T11:14:55.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b637fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9b641457f0 0x7f9b64138090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:53921/0 (socket says 172.21.15.157:53921) 2024-03-29T11:14:55.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b637fe640 1 -- 172.21.15.157:0/2003192761 learned_addr learned my addr 172.21.15.157:0/2003192761 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:14:55.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9b640a9d70 2024-03-29T11:14:55.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:55.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9b641457f0 2024-03-29T11:14:55.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:55.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081524594 0 0) 0x7f9b641387a0 con 0x7f9b641457f0 2024-03-29T11:14:55.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:55.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:55.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:55.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b4b7fe640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:14:55.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.395+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b50004eb0 con 0x7f9b641457f0 2024-03-29T11:14:55.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9b6413f650 2024-03-29T11:14:55.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 464776457 0 0) 0x7f9b50004eb0 con 0x7f9b641457f0 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b641387a0 con 0x7f9b641457f0 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226636590 0 0) 0x7f9b640ace00 con 0x7f9b640a9d70 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b50004eb0 con 0x7f9b640a9d70 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423826573 0 0) 0x7f9b641d70d0 con 0x7f9b6413f650 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b640ace00 con 0x7f9b6413f650 2024-03-29T11:14:55.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9b58001850 con 0x7f9b641457f0 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): dump: 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:55.384 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 974465554 0 0) 0x7f9b641387a0 con 0x7f9b641457f0 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 >> v1:172.21.15.157:6790/0 conn(0x7f9b640a9d70 legacy=0x7f9b641d75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 >> v1:172.21.15.195:6789/0 conn(0x7f9b6413f650 legacy=0x7f9b64137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 monclient: found mon.a 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b640ad0e0 con 0x7f9b641457f0 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.399+0000 7f9b4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b6973d640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b6973d640 10 monclient: _renew_subs 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b6973d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:14:55.385 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b6973d640 1 -- 172.21.15.157:0/2003192761 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6413fda0 con 0x7f9b641457f0 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b6973d640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b6973d640 10 monclient: _renew_subs 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b6973d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b6973d640 1 -- 172.21.15.157:0/2003192761 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b641d7ea0 con 0x7f9b641457f0 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b6973d640 1 librados: init done 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b580019b0 con 0x7f9b641457f0 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9b58003870 con 0x7f9b641457f0 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b617fa640 4 set_mon_vals no callback set 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.400+0000 7f9b4b7fe640 10 monclient: dump: 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:14:55.386 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.401+0000 7f9b6973d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.401+0000 7f9b6973d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.401+0000 7f9b6973d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.401+0000 7f9b6973d640 1 -- 172.21.15.157:0/2003192761 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b44005080 con 0x7f9b641457f0 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.403+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f9b580039d0 con 0x7f9b641457f0 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.403+0000 7f9b62ffd640 10 monclient: get_auth_request con 0x7f9b5006dcb0 auth_method 0 2024-03-29T11:14:55.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.404+0000 7f9b4b7fe640 10 client.4359.objecter ms_handle_connect 0x7f9b5006dcb0 2024-03-29T11:14:55.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.404+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f9b58084990 con 0x7f9b641457f0 2024-03-29T11:14:55.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.404+0000 7f9b4b7fe640 10 client.4359.objecter ms_dispatch 0x7f9b64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:55.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.404+0000 7f9b4b7fe640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:55.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.404+0000 7f9b4b7fe640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:55.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.404+0000 7f9b4b7fe640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-29T11:14:55.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.410+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b580500a0 con 0x7f9b641457f0 2024-03-29T11:14:55.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.411+0000 7f9b4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:55.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.411+0000 7f9b4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:55.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.733+0000 7f9b6973d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:55.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.733+0000 7f9b6973d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:55.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.733+0000 7f9b6973d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:14:55.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.733+0000 7f9b6973d640 1 -- 172.21.15.157:0/2003192761 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b44005330 con 0x7f9b641457f0 2024-03-29T11:14:55.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.738+0000 7f9b4b7fe640 1 -- 172.21.15.157:0/2003192761 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f9b580521a0 con 0x7f9b641457f0 2024-03-29T11:14:55.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.738+0000 7f9b4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:55.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.738+0000 7f9b4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:55.717 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:14:55.717 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:55.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.742+0000 7f9b6973d640 10 librados: watch_flush enter 2024-03-29T11:14:55.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.742+0000 7f9b6973d640 10 librados: watch_flush exit 2024-03-29T11:14:55.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.742+0000 7f9b6973d640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-29T11:14:55.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.742+0000 7f9b6973d640 10 client.4359.objecter successfully canceled tick 2024-03-29T11:14:55.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.742+0000 7f9b6973d640 1 -- 172.21.15.157:0/2003192761 >> v1:172.21.15.157:6816/1747058247 conn(0x7f9b5006dcb0 legacy=0x7f9b50070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:55.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.742+0000 7f9b6973d640 10 monclient: shutdown 2024-03-29T11:14:55.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.742+0000 7f9b6973d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:55.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.742+0000 7f9b6973d640 1 -- 172.21.15.157:0/2003192761 >> v1:172.21.15.157:6789/0 conn(0x7f9b641457f0 legacy=0x7f9b64138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:55.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.742+0000 7f9b6973d640 1 -- 172.21.15.157:0/2003192761 shutdown_connections 2024-03-29T11:14:55.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.742+0000 7f9b6973d640 1 -- 172.21.15.157:0/2003192761 >> 172.21.15.157:0/2003192761 conn(0x7f9b64066a00 msgr2=0x7f9b640b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:55.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.743+0000 7f9b6973d640 1 -- 172.21.15.157:0/2003192761 shutdown_connections 2024-03-29T11:14:55.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.743+0000 7f9b6973d640 1 -- 172.21.15.157:0/2003192761 wait complete. 2024-03-29T11:14:55.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:14:55.743+0000 7f9b6973d640 1 librados: shutdown 2024-03-29T11:14:55.738 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:02.548 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:02.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.724+0000 7f3167104640 10 monclient: get_monmap_and_config 2024-03-29T11:15:02.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.724+0000 7f3167104640 10 monclient: build_initial_monmap 2024-03-29T11:15:02.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.724+0000 7f3167104640 10 monclient: monmap: 2024-03-29T11:15:02.702 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:02.703 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:02.703 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:15:02.725988+0000 2024-03-29T11:15:02.703 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:15:02.725988+0000 2024-03-29T11:15:02.703 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:02.703 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:02.703 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:02.703 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:15:02.703 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:15:02.703 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:02.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.725+0000 7f3167104640 1 Processor -- start 2024-03-29T11:15:02.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.726+0000 7f3167104640 1 -- start start 2024-03-29T11:15:02.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.726+0000 7f3167104640 10 monclient: init 2024-03-29T11:15:02.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f3167104640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:02.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f3167104640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:02.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f3167104640 10 monclient(hunting): picked mon.noname-b con 0x7f31601422b0 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:02.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f3167104640 10 monclient(hunting): picked mon.noname-a con 0x7f3160142690 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:02.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f3167104640 10 monclient(hunting): picked mon.noname-c con 0x7f31600b17d0 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:02.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f3167104640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3160142c80 con 0x7f3160142690 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f3167104640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600aae40 con 0x7f31600b17d0 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f3167104640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f316013fbd0 con 0x7f31601422b0 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f3167104640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f3167104640 10 monclient(hunting): authenticate will time out at 1114.924012s 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3160142690 0x7f31600b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:44322/0 (socket says 172.21.15.157:44322) 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315ffff640 1 -- 172.21.15.157:0/1816413716 learned_addr learned my addr 172.21.15.157:0/1816413716 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532606770 0 0) 0x7f316013fbd0 con 0x7f31601422b0 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3150003520 con 0x7f31601422b0 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807821601 0 0) 0x7f3160142c80 con 0x7f3160142690 2024-03-29T11:15:02.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:02.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:02.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:02.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.727+0000 7f315f7fe640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:15:02.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f316013fbd0 con 0x7f3160142690 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654794095 0 0) 0x7f31600aae40 con 0x7f31600b17d0 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3160142c80 con 0x7f31600b17d0 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 172926273 0 0) 0x7f3150003520 con 0x7f31601422b0 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 >> v1:172.21.15.195:6789/0 conn(0x7f31600b17d0 legacy=0x7f31600b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 >> v1:172.21.15.157:6789/0 conn(0x7f3160142690 legacy=0x7f31600b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:15:02.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:02.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:15:02.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3160151940 con 0x7f31601422b0 2024-03-29T11:15:02.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:02.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:02.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:02.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3154001690 con 0x7f31601422b0 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: dump: 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f3167104640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:15:02.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f3167104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3154001870 con 0x7f31601422b0 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 1 -- 172.21.15.157:0/1816413716 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3154001c00 con 0x7f31601422b0 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f315f7fe640 10 monclient: dump: 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:02.711 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f3167104640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.728+0000 7f3167104640 4 set_mon_vals no callback set 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.729+0000 7f3167104640 10 monclient: shutdown 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.729+0000 7f3167104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.729+0000 7f3167104640 1 -- 172.21.15.157:0/1816413716 >> v1:172.21.15.157:6790/0 conn(0x7f31601422b0 legacy=0x7f316013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.729+0000 7f3167104640 1 -- 172.21.15.157:0/1816413716 shutdown_connections 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.729+0000 7f3167104640 1 -- 172.21.15.157:0/1816413716 >> 172.21.15.157:0/1816413716 conn(0x7f3160066a00 msgr2=0x7f3160066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.729+0000 7f3167104640 1 -- 172.21.15.157:0/1816413716 shutdown_connections 2024-03-29T11:15:02.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.729+0000 7f3167104640 1 -- 172.21.15.157:0/1816413716 wait complete. 2024-03-29T11:15:02.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.730+0000 7f3167104640 10 monclient: build_initial_monmap 2024-03-29T11:15:02.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.730+0000 7f3167104640 10 monclient: monmap: 2024-03-29T11:15:02.721 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:02.721 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:02.721 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:15:02.721 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:15:02.721 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:02.721 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:02.721 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.731+0000 7f3167104640 1 librados: starting msgr at 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.731+0000 7f3167104640 1 librados: starting objecter 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.731+0000 7f3167104640 1 Processor -- start 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.732+0000 7f3167104640 1 -- start start 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.732+0000 7f3167104640 1 librados: setting wanted keys 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.732+0000 7f3167104640 1 librados: calling monclient init 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.732+0000 7f3167104640 10 monclient: init 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.733+0000 7f3167104640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.733+0000 7f3167104640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.733+0000 7f3167104640 10 monclient(hunting): picked mon.noname-c con 0x7f31600b17d0 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:02.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.733+0000 7f3167104640 10 monclient(hunting): picked mon.noname-a con 0x7f31601422b0 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.733+0000 7f3164ea0640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f31600b17d0 0x7f316014dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:45274/0 (socket says 172.21.15.157:45274) 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.733+0000 7f3164ea0640 1 -- 172.21.15.157:0/1254185541 learned_addr learned my addr 172.21.15.157:0/1254185541 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.733+0000 7f3167104640 10 monclient(hunting): picked mon.noname-b con 0x7f3160142690 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.734+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600b13d0 con 0x7f31601422b0 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.733+0000 7f315d7fa640 10 client.?.objecter ms_handle_connect 0x7f31600b17d0 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.734+0000 7f315d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.734+0000 7f315d7fa640 10 client.?.objecter ms_handle_connect 0x7f31601422b0 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.734+0000 7f315d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.734+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3160143360 con 0x7f3160142690 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.734+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31600af410 con 0x7f31600b17d0 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.734+0000 7f3167104640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.734+0000 7f3167104640 10 monclient(hunting): authenticate will time out at 1114.931533s 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.734+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432098610 0 0) 0x7f31600af410 con 0x7f31600b17d0 2024-03-29T11:15:02.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3144004eb0 con 0x7f31600b17d0 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378332112 0 0) 0x7f31600b13d0 con 0x7f31601422b0 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31600af410 con 0x7f31601422b0 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 client.?.objecter ms_handle_connect 0x7f3160142690 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 213539900 0 0) 0x7f3144004eb0 con 0x7f31600b17d0 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31600b13d0 con 0x7f31600b17d0 2024-03-29T11:15:02.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3154001c40 con 0x7f31600b17d0 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.735+0000 7f315d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 10 monclient(hunting): dump: 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:02.725 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:02.726 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:02.726 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:02.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:02.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2025750015 0 0) 0x7f31600af410 con 0x7f31601422b0 2024-03-29T11:15:02.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3144004eb0 con 0x7f31601422b0 2024-03-29T11:15:02.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160148044 0 0) 0x7f3160143360 con 0x7f3160142690 2024-03-29T11:15:02.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:02.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:02.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:02.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:15:02.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31600af410 con 0x7f3160142690 2024-03-29T11:15:02.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f314c0016b0 con 0x7f31601422b0 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 10 monclient(hunting): dump: 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:02.728 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4077409951 0 0) 0x7f31600b13d0 con 0x7f31600b17d0 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 >> v1:172.21.15.195:6789/0 conn(0x7f3160142690 legacy=0x7f31601e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.736+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 >> v1:172.21.15.157:6789/0 conn(0x7f31601422b0 legacy=0x7f31600af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f315d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f315d7fa640 1 monclient: found mon.c 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f315d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f315d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31600af050 con 0x7f31600b17d0 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f315d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f315d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f315d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f3167104640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:15:02.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f3167104640 10 monclient: _renew_subs 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f3167104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316013c430 con 0x7f31600b17d0 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f3167104640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f3167104640 10 monclient: _renew_subs 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f3167104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f316013bbd0 con 0x7f31600b17d0 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.737+0000 7f3167104640 1 librados: init done 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.738+0000 7f3167104640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.738+0000 7f3167104640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.738+0000 7f3167104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.738+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31600af240 con 0x7f31600b17d0 2024-03-29T11:15:02.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.739+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31540011f0 con 0x7f31600b17d0 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.739+0000 7f315d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.739+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3154003b60 con 0x7f31600b17d0 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.739+0000 7f315d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.739+0000 7f315d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.739+0000 7f315d7fa640 10 monclient: dump: 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:02.731 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:02.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.739+0000 7f315effd640 4 set_mon_vals no callback set 2024-03-29T11:15:02.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.741+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f3154003d80 con 0x7f31600b17d0 2024-03-29T11:15:02.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.741+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f3154084bb0 con 0x7f31600b17d0 2024-03-29T11:15:02.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.741+0000 7f315d7fa640 10 client.4370.objecter ms_dispatch 0x7f3160000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:02.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.741+0000 7f315d7fa640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:02.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.741+0000 7f315d7fa640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:02.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.742+0000 7f315ffff640 10 monclient: get_auth_request con 0x7f314406df80 auth_method 0 2024-03-29T11:15:02.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.742+0000 7f315d7fa640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-29T11:15:02.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.742+0000 7f315d7fa640 10 client.4370.objecter ms_handle_connect 0x7f314406df80 2024-03-29T11:15:02.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.747+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3154050490 con 0x7f31600b17d0 2024-03-29T11:15:02.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.748+0000 7f315d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:02.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:02.748+0000 7f315d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:03.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.109+0000 7f3167104640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:03.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.109+0000 7f3167104640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:03.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.109+0000 7f3167104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:03.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.109+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f316013c670 con 0x7f31600b17d0 2024-03-29T11:15:03.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.110+0000 7f315d7fa640 1 -- 172.21.15.157:0/1254185541 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f31540523c0 con 0x7f31600b17d0 2024-03-29T11:15:03.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.110+0000 7f315d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:03.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.110+0000 7f315d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:03.089 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:15:03.089 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:03.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.113+0000 7f3167104640 10 librados: watch_flush enter 2024-03-29T11:15:03.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.114+0000 7f3167104640 10 librados: watch_flush exit 2024-03-29T11:15:03.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.114+0000 7f3167104640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-29T11:15:03.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.114+0000 7f3167104640 10 client.4370.objecter successfully canceled tick 2024-03-29T11:15:03.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.114+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 >> v1:172.21.15.157:6816/1747058247 conn(0x7f314406df80 legacy=0x7f3144070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:03.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.114+0000 7f3167104640 10 monclient: shutdown 2024-03-29T11:15:03.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.114+0000 7f3167104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:03.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.114+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 >> v1:172.21.15.157:6790/0 conn(0x7f31600b17d0 legacy=0x7f316014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:03.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.114+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 shutdown_connections 2024-03-29T11:15:03.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.114+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 >> 172.21.15.157:0/1254185541 conn(0x7f3160066a00 msgr2=0x7f31600b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:03.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.114+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 shutdown_connections 2024-03-29T11:15:03.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.115+0000 7f3167104640 1 -- 172.21.15.157:0/1254185541 wait complete. 2024-03-29T11:15:03.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:03.115+0000 7f3167104640 1 librados: shutdown 2024-03-29T11:15:03.108 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:09.916 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:10.056 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.078+0000 7f536e801640 10 monclient: get_monmap_and_config 2024-03-29T11:15:10.056 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.078+0000 7f536e801640 10 monclient: build_initial_monmap 2024-03-29T11:15:10.057 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.078+0000 7f536e801640 10 monclient: monmap: 2024-03-29T11:15:10.057 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:10.057 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:10.057 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:15:10.079644+0000 2024-03-29T11:15:10.057 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:15:10.079644+0000 2024-03-29T11:15:10.057 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:10.057 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:10.057 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:10.057 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:15:10.058 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:15:10.058 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:10.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.079+0000 7f536e801640 1 Processor -- start 2024-03-29T11:15:10.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.080+0000 7f536e801640 1 -- start start 2024-03-29T11:15:10.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.080+0000 7f536e801640 10 monclient: init 2024-03-29T11:15:10.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.081+0000 7f536e801640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:10.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.081+0000 7f536e801640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:10.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.081+0000 7f536e801640 10 monclient(hunting): picked mon.noname-a con 0x7f53681457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:10.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.081+0000 7f536e801640 10 monclient(hunting): picked mon.noname-c con 0x7f536813f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:10.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.081+0000 7f536e801640 10 monclient(hunting): picked mon.noname-b con 0x7f53680a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:10.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.081+0000 7f536e801640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53680aaef0 con 0x7f53681457f0 2024-03-29T11:15:10.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.081+0000 7f536e801640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5368145bd0 con 0x7f536813f650 2024-03-29T11:15:10.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.081+0000 7f536e801640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5368145eb0 con 0x7f53680a9d70 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.081+0000 7f536e801640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.081+0000 7f536e801640 10 monclient(hunting): authenticate will time out at 1122.278527s 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.082+0000 7f53677fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f536813f650 0x7f53680b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:53534/0 (socket says 172.21.15.157:53534) 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.082+0000 7f53677fe640 1 -- 172.21.15.157:0/1339607700 learned_addr learned my addr 172.21.15.157:0/1339607700 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.082+0000 7f5366ffd640 1 -- 172.21.15.157:0/1339607700 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30059726 0 0) 0x7f5368145eb0 con 0x7f53680a9d70 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.082+0000 7f5366ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.083+0000 7f5366ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.083+0000 7f5366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.083+0000 7f5366ffd640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.083+0000 7f5366ffd640 1 -- 172.21.15.157:0/1339607700 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5350003520 con 0x7f53680a9d70 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.083+0000 7f5366ffd640 1 -- 172.21.15.157:0/1339607700 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3675286793 0 0) 0x7f5350003520 con 0x7f53680a9d70 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.083+0000 7f5366ffd640 1 -- 172.21.15.157:0/1339607700 >> v1:172.21.15.195:6789/0 conn(0x7f536813f650 legacy=0x7f53680b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.083+0000 7f5366ffd640 1 -- 172.21.15.157:0/1339607700 >> v1:172.21.15.157:6789/0 conn(0x7f53681457f0 legacy=0x7f536813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:15:10.067 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 1 monclient: found mon.noname-b 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 1 -- 172.21.15.157:0/1339607700 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5368146340 con 0x7f53680a9d70 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 10 monclient: _finish_auth 0 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 1 -- 172.21.15.157:0/1339607700 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5360001a10 con 0x7f53680a9d70 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.084+0000 7f5366ffd640 10 monclient: dump: 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:10.068 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.085+0000 7f5366ffd640 1 -- 172.21.15.157:0/1339607700 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5360001d20 con 0x7f53680a9d70 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.085+0000 7f5366ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.085+0000 7f5366ffd640 1 -- 172.21.15.157:0/1339607700 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f53600020b0 con 0x7f53680a9d70 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.085+0000 7f5366ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.085+0000 7f5366ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.085+0000 7f5366ffd640 10 monclient: dump: 2024-03-29T11:15:10.069 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.085+0000 7f536e801640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.085+0000 7f536e801640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.085+0000 7f536e801640 4 set_mon_vals no callback set 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.087+0000 7f536e801640 10 monclient: shutdown 2024-03-29T11:15:10.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.087+0000 7f536e801640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:10.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.087+0000 7f536e801640 1 -- 172.21.15.157:0/1339607700 >> v1:172.21.15.157:6790/0 conn(0x7f53680a9d70 legacy=0x7f536814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:10.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.087+0000 7f536e801640 1 -- 172.21.15.157:0/1339607700 shutdown_connections 2024-03-29T11:15:10.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.087+0000 7f536e801640 1 -- 172.21.15.157:0/1339607700 >> 172.21.15.157:0/1339607700 conn(0x7f5368066a00 msgr2=0x7f5368066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:10.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.088+0000 7f536e801640 1 -- 172.21.15.157:0/1339607700 shutdown_connections 2024-03-29T11:15:10.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.088+0000 7f536e801640 1 -- 172.21.15.157:0/1339607700 wait complete. 2024-03-29T11:15:10.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.089+0000 7f536e801640 10 monclient: build_initial_monmap 2024-03-29T11:15:10.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.090+0000 7f536e801640 10 monclient: monmap: 2024-03-29T11:15:10.087 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:10.087 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:10.087 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:15:10.087 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.090+0000 7f536e801640 1 librados: starting msgr at 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.090+0000 7f536e801640 1 librados: starting objecter 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.090+0000 7f536e801640 1 Processor -- start 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.091+0000 7f536e801640 1 -- start start 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.091+0000 7f536e801640 1 librados: setting wanted keys 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.091+0000 7f536e801640 1 librados: calling monclient init 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.091+0000 7f536e801640 10 monclient: init 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.092+0000 7f536e801640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:10.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.092+0000 7f536e801640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.092+0000 7f536e801640 10 monclient(hunting): picked mon.noname-a con 0x7f53680a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.092+0000 7f536e801640 10 monclient(hunting): picked mon.noname-c con 0x7f536813f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.094+0000 7f53677fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f536813f650 0x7f53681da740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:58544/0 (socket says 172.21.15.157:58544) 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.094+0000 7f536e801640 10 monclient(hunting): picked mon.noname-b con 0x7f53681457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.094+0000 7f53677fe640 1 -- 172.21.15.157:0/2089500549 learned_addr learned my addr 172.21.15.157:0/2089500549 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.094+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5368144f20 con 0x7f53680a9d70 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.094+0000 7f5364ff9640 10 client.?.objecter ms_handle_connect 0x7f53681457f0 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.094+0000 7f5364ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.094+0000 7f5364ff9640 10 client.?.objecter ms_handle_connect 0x7f536813f650 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.094+0000 7f5364ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.094+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53681db560 con 0x7f53681457f0 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5368141370 con 0x7f536813f650 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f536e801640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:10.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f536e801640 10 monclient(hunting): authenticate will time out at 1122.292054s 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394787010 0 0) 0x7f5368141370 con 0x7f536813f650 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5354004eb0 con 0x7f536813f650 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206470251 0 0) 0x7f53681db560 con 0x7f53681457f0 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5368141370 con 0x7f53681457f0 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 client.?.objecter ms_handle_connect 0x7f53680a9d70 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:10.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153338197 0 0) 0x7f5368144f20 con 0x7f53680a9d70 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.095+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53681db560 con 0x7f53680a9d70 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 902114260 0 0) 0x7f53681db560 con 0x7f53680a9d70 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5368144f20 con 0x7f53680a9d70 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f535c0016f0 con 0x7f53680a9d70 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient(hunting): dump: 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:10.091 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 605509368 0 0) 0x7f5354004eb0 con 0x7f536813f650 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53681db560 con 0x7f536813f650 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5358001490 con 0x7f536813f650 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient(hunting): dump: 2024-03-29T11:15:10.092 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:10.093 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:10.093 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:10.093 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:10.093 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:10.093 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:10.093 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:10.093 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:10.094 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:10.094 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:10.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2227314569 0 0) 0x7f5368144f20 con 0x7f53680a9d70 2024-03-29T11:15:10.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 >> v1:172.21.15.157:6790/0 conn(0x7f536813f650 legacy=0x7f53681da740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:10.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 >> v1:172.21.15.195:6789/0 conn(0x7f53681457f0 legacy=0x7f53681dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:10.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:15:10.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 monclient: found mon.a 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5368141650 con 0x7f53680a9d70 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient: _finish_auth 0 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f536e801640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f536e801640 10 monclient: _renew_subs 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f535c001be0 con 0x7f53680a9d70 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f536e801640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5368144620 con 0x7f53680a9d70 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f536e801640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f536e801640 10 monclient: _renew_subs 2024-03-29T11:15:10.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f536e801640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f536813c610 con 0x7f53680a9d70 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f536e801640 1 librados: init done 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f535c003790 con 0x7f53680a9d70 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5364ff9640 10 monclient: dump: 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:10.096 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.096+0000 7f5365ffb640 4 set_mon_vals no callback set 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.098+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f535c019760 con 0x7f53680a9d70 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.099+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f535c0835d0 con 0x7f53680a9d70 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.099+0000 7f5364ff9640 10 client.4368.objecter ms_dispatch 0x7f5368000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.099+0000 7f5364ff9640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.099+0000 7f5364ff9640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.099+0000 7f536e801640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.099+0000 7f536e801640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.099+0000 7f536e801640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.099+0000 7f5364ff9640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.099+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5368075f10 con 0x7f53680a9d70 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.099+0000 7f53677fe640 10 monclient: get_auth_request con 0x7f535406ded0 auth_method 0 2024-03-29T11:15:10.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.100+0000 7f5364ff9640 10 client.4368.objecter ms_handle_connect 0x7f535406ded0 2024-03-29T11:15:10.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.107+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f535c04ef10 con 0x7f53680a9d70 2024-03-29T11:15:10.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.107+0000 7f5364ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:10.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.107+0000 7f5364ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:10.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.430+0000 7f536e801640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:10.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.430+0000 7f536e801640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:10.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.430+0000 7f536e801640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:10.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.430+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53680b3680 con 0x7f53680a9d70 2024-03-29T11:15:10.410 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.432+0000 7f5364ff9640 1 -- 172.21.15.157:0/2089500549 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f535c050e40 con 0x7f53680a9d70 2024-03-29T11:15:10.410 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.432+0000 7f5364ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:10.410 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.432+0000 7f5364ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:10.411 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:15:10.411 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:10.414 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.436+0000 7f536e801640 10 librados: watch_flush enter 2024-03-29T11:15:10.414 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.436+0000 7f536e801640 10 librados: watch_flush exit 2024-03-29T11:15:10.415 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.436+0000 7f536e801640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:15:10.415 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.436+0000 7f536e801640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:15:10.415 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.437+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 >> v1:172.21.15.157:6816/1747058247 conn(0x7f535406ded0 legacy=0x7f5354070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:10.415 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.437+0000 7f536e801640 10 monclient: shutdown 2024-03-29T11:15:10.415 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.437+0000 7f536e801640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:10.415 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.437+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 >> v1:172.21.15.157:6789/0 conn(0x7f53680a9d70 legacy=0x7f536813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:10.415 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.437+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 shutdown_connections 2024-03-29T11:15:10.415 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.437+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 >> 172.21.15.157:0/2089500549 conn(0x7f5368066a00 msgr2=0x7f53680b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:10.416 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.439+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 shutdown_connections 2024-03-29T11:15:10.416 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.439+0000 7f536e801640 1 -- 172.21.15.157:0/2089500549 wait complete. 2024-03-29T11:15:10.416 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:10.439+0000 7f536e801640 1 librados: shutdown 2024-03-29T11:15:10.433 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:17.241 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:17.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.401+0000 7f66c32c8640 10 monclient: get_monmap_and_config 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.401+0000 7f66c32c8640 10 monclient: build_initial_monmap 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.401+0000 7f66c32c8640 10 monclient: monmap: 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:15:17.402926+0000 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:15:17.402926+0000 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:15:17.380 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:17.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.402+0000 7f66c32c8640 1 Processor -- start 2024-03-29T11:15:17.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.402+0000 7f66c32c8640 1 -- start start 2024-03-29T11:15:17.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.402+0000 7f66c32c8640 10 monclient: init 2024-03-29T11:15:17.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c32c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:17.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c32c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c32c8640 10 monclient(hunting): picked mon.noname-a con 0x7f66bc0a9750 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c32c8640 10 monclient(hunting): picked mon.noname-c con 0x7f66bc13e1e0 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c32c8640 10 monclient(hunting): picked mon.noname-b con 0x7f66bc071c30 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c32c8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66bc0ab300 con 0x7f66bc0a9750 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c32c8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66bc0ac2f0 con 0x7f66bc13e1e0 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c32c8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66bc0ad8d0 con 0x7f66bc071c30 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c32c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c32c8640 10 monclient(hunting): authenticate will time out at 1129.600573s 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c1865640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f66bc071c30 0x7f66bc0ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:42308/0 (socket says 172.21.15.157:42308) 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.403+0000 7f66c1865640 1 -- 172.21.15.157:0/682517011 learned_addr learned my addr 172.21.15.157:0/682517011 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.404+0000 7f66b3fff640 1 -- 172.21.15.157:0/682517011 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114036439 0 0) 0x7f66bc0ac2f0 con 0x7f66bc13e1e0 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.404+0000 7f66b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:17.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.404+0000 7f66b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:17.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.404+0000 7f66b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:17.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.404+0000 7f66b3fff640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:15:17.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.404+0000 7f66b3fff640 1 -- 172.21.15.157:0/682517011 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a8004630 con 0x7f66bc13e1e0 2024-03-29T11:15:17.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.404+0000 7f66b3fff640 1 -- 172.21.15.157:0/682517011 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3146544625 0 0) 0x7f66a8004630 con 0x7f66bc13e1e0 2024-03-29T11:15:17.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.404+0000 7f66b3fff640 1 -- 172.21.15.157:0/682517011 >> v1:172.21.15.157:6790/0 conn(0x7f66bc071c30 legacy=0x7f66bc0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:17.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.404+0000 7f66b3fff640 1 -- 172.21.15.157:0/682517011 >> v1:172.21.15.157:6789/0 conn(0x7f66bc0a9750 legacy=0x7f66bc13dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 1 monclient: found mon.noname-c 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 1 -- 172.21.15.157:0/682517011 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc0ada60 con 0x7f66bc13e1e0 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 10 monclient: _finish_auth 0 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 1 -- 172.21.15.157:0/682517011 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f66ac001690 con 0x7f66bc13e1e0 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:17.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66b3fff640 10 monclient: dump: 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66c32c8640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.406+0000 7f66c32c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.407+0000 7f66b3fff640 1 -- 172.21.15.157:0/682517011 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66ac001920 con 0x7f66bc13e1e0 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.407+0000 7f66b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:17.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.407+0000 7f66b3fff640 1 -- 172.21.15.157:0/682517011 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f66ac001d30 con 0x7f66bc13e1e0 2024-03-29T11:15:17.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.407+0000 7f66b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:17.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.407+0000 7f66b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:17.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.407+0000 7f66b3fff640 10 monclient: dump: 2024-03-29T11:15:17.388 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:17.388 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.407+0000 7f66c32c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.407+0000 7f66c32c8640 4 set_mon_vals no callback set 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.408+0000 7f66c32c8640 10 monclient: shutdown 2024-03-29T11:15:17.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.408+0000 7f66c32c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:17.390 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.408+0000 7f66c32c8640 1 -- 172.21.15.157:0/682517011 >> v1:172.21.15.195:6789/0 conn(0x7f66bc13e1e0 legacy=0x7f66bc0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:17.390 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.408+0000 7f66c32c8640 1 -- 172.21.15.157:0/682517011 shutdown_connections 2024-03-29T11:15:17.390 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.408+0000 7f66c32c8640 1 -- 172.21.15.157:0/682517011 >> 172.21.15.157:0/682517011 conn(0x7f66bc06b660 msgr2=0x7f66bc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:17.390 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.408+0000 7f66c32c8640 1 -- 172.21.15.157:0/682517011 shutdown_connections 2024-03-29T11:15:17.390 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.408+0000 7f66c32c8640 1 -- 172.21.15.157:0/682517011 wait complete. 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.409+0000 7f66c32c8640 10 monclient: build_initial_monmap 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.409+0000 7f66c32c8640 10 monclient: monmap: 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:15:17.399 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.409+0000 7f66c32c8640 1 librados: starting msgr at 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.409+0000 7f66c32c8640 1 librados: starting objecter 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.409+0000 7f66c32c8640 1 Processor -- start 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.409+0000 7f66c32c8640 1 -- start start 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.409+0000 7f66c32c8640 1 librados: setting wanted keys 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.409+0000 7f66c32c8640 1 librados: calling monclient init 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.409+0000 7f66c32c8640 10 monclient: init 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.410+0000 7f66c32c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.410+0000 7f66c32c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.410+0000 7f66c32c8640 10 monclient(hunting): picked mon.noname-a con 0x7f66bc071c30 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66c32c8640 10 monclient(hunting): picked mon.noname-c con 0x7f66bc0a9750 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66c32c8640 10 monclient(hunting): picked mon.noname-b con 0x7f66bc13e1e0 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66c32c8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66bc140eb0 con 0x7f66bc071c30 2024-03-29T11:15:17.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66c32c8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66bc13f830 con 0x7f66bc13e1e0 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66c32c8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66bc13fb10 con 0x7f66bc0a9750 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66c32c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66c32c8640 10 monclient(hunting): authenticate will time out at 1129.608508s 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66c0863640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f66bc0a9750 0x7f66bc1df830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:44654/0 (socket says 172.21.15.157:44654) 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66c0863640 1 -- 172.21.15.157:0/525361938 learned_addr learned my addr 172.21.15.157:0/525361938 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66b1ffb640 10 client.?.objecter ms_handle_connect 0x7f66bc0a9750 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.411+0000 7f66b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 client.?.objecter ms_handle_connect 0x7f66bc13e1e0 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219954619 0 0) 0x7f66bc13fb10 con 0x7f66bc0a9750 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:17.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a0004eb0 con 0x7f66bc0a9750 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027518265 0 0) 0x7f66bc13f830 con 0x7f66bc13e1e0 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66bc13fb10 con 0x7f66bc13e1e0 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3531595691 0 0) 0x7f66a0004eb0 con 0x7f66bc0a9750 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66bc13f830 con 0x7f66bc0a9750 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f66ac001710 con 0x7f66bc0a9750 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): dump: 2024-03-29T11:15:17.402 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3365740809 0 0) 0x7f66bc13fb10 con 0x7f66bc13e1e0 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66a0004eb0 con 0x7f66bc13e1e0 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f66b4001ba0 con 0x7f66bc13e1e0 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:17.403 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:17.404 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 10 monclient(hunting): dump: 2024-03-29T11:15:17.404 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:17.404 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:17.404 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:17.404 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:17.404 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:17.404 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:17.404 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1227447732 0 0) 0x7f66bc13f830 con 0x7f66bc0a9750 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 >> v1:172.21.15.195:6789/0 conn(0x7f66bc13e1e0 legacy=0x7f66bc1e64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.412+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 >> v1:172.21.15.157:6789/0 conn(0x7f66bc071c30 legacy=0x7f66bc1df120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 1 monclient: found mon.c 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc141100 con 0x7f66bc0a9750 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:17.405 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66c32c8640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66c32c8640 10 monclient: _renew_subs 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66c32c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66c32c8640 1 -- 172.21.15.157:0/525361938 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66bc1413e0 con 0x7f66bc0a9750 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66c32c8640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66ac0038a0 con 0x7f66bc0a9750 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f66ac003c30 con 0x7f66bc0a9750 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b1ffb640 10 monclient: dump: 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:17.406 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.413+0000 7f66b37fe640 4 set_mon_vals no callback set 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.414+0000 7f66c32c8640 10 monclient: _renew_subs 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.414+0000 7f66c32c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.414+0000 7f66c32c8640 1 -- 172.21.15.157:0/525361938 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66bc138aa0 con 0x7f66bc0a9750 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.415+0000 7f66c32c8640 1 librados: init done 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.415+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f66ac003d90 con 0x7f66bc0a9750 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f66ac084b00 con 0x7f66bc0a9750 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66b1ffb640 10 client.4379.objecter ms_dispatch 0x7f66bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:17.407 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66b1ffb640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:17.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66b1ffb640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:17.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66b1ffb640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T11:15:17.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66c1064640 10 monclient: get_auth_request con 0x7f66a006de40 auth_method 0 2024-03-29T11:15:17.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66c32c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:17.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66c32c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:17.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66c32c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:17.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66c32c8640 1 -- 172.21.15.157:0/525361938 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66bc075f10 con 0x7f66bc0a9750 2024-03-29T11:15:17.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.416+0000 7f66b1ffb640 10 client.4379.objecter ms_handle_connect 0x7f66a006de40 2024-03-29T11:15:17.408 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.428+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66ac050580 con 0x7f66bc0a9750 2024-03-29T11:15:17.409 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.428+0000 7f66b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:17.409 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.428+0000 7f66b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:17.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.733+0000 7f66c32c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:17.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.733+0000 7f66c32c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:17.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.733+0000 7f66c32c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:17.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.733+0000 7f66c32c8640 1 -- 172.21.15.157:0/525361938 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66bc04ef90 con 0x7f66bc0a9750 2024-03-29T11:15:17.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.734+0000 7f66b1ffb640 1 -- 172.21.15.157:0/525361938 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f66ac0841b0 con 0x7f66bc0a9750 2024-03-29T11:15:17.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.734+0000 7f66b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:17.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.734+0000 7f66b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:17.712 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:15:17.712 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:17.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.737+0000 7f66c32c8640 10 librados: watch_flush enter 2024-03-29T11:15:17.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.737+0000 7f66c32c8640 10 librados: watch_flush exit 2024-03-29T11:15:17.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.737+0000 7f66c32c8640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T11:15:17.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.737+0000 7f66c32c8640 10 client.4379.objecter successfully canceled tick 2024-03-29T11:15:17.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.738+0000 7f66c32c8640 1 -- 172.21.15.157:0/525361938 >> v1:172.21.15.157:6816/1747058247 conn(0x7f66a006de40 legacy=0x7f66a0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:17.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.738+0000 7f66c32c8640 10 monclient: shutdown 2024-03-29T11:15:17.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.738+0000 7f66c32c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:17.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.738+0000 7f66c32c8640 1 -- 172.21.15.157:0/525361938 >> v1:172.21.15.157:6790/0 conn(0x7f66bc0a9750 legacy=0x7f66bc1df830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:17.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.738+0000 7f66c32c8640 1 -- 172.21.15.157:0/525361938 shutdown_connections 2024-03-29T11:15:17.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.738+0000 7f66c32c8640 1 -- 172.21.15.157:0/525361938 >> 172.21.15.157:0/525361938 conn(0x7f66bc06b660 msgr2=0x7f66bc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:17.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.738+0000 7f66c32c8640 1 -- 172.21.15.157:0/525361938 shutdown_connections 2024-03-29T11:15:17.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.738+0000 7f66c32c8640 1 -- 172.21.15.157:0/525361938 wait complete. 2024-03-29T11:15:17.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:17.738+0000 7f66c32c8640 1 librados: shutdown 2024-03-29T11:15:17.732 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:24.539 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:24.697 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.719+0000 7f6800882640 10 monclient: get_monmap_and_config 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.719+0000 7f6800882640 10 monclient: build_initial_monmap 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.719+0000 7f6800882640 10 monclient: monmap: 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:15:24.721054+0000 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:15:24.721054+0000 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:15:24.698 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:24.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.723+0000 7f6800882640 1 Processor -- start 2024-03-29T11:15:24.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.723+0000 7f6800882640 1 -- start start 2024-03-29T11:15:24.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.723+0000 7f6800882640 10 monclient: init 2024-03-29T11:15:24.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.724+0000 7f6800882640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:24.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.724+0000 7f6800882640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:24.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.724+0000 7f6800882640 10 monclient(hunting): picked mon.noname-c con 0x7f67fc1457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:24.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.729+0000 7f67fa59c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f67fc1457f0 0x7f67fc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:42329/0 (socket says 172.21.15.157:42329) 2024-03-29T11:15:24.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.729+0000 7f6800882640 10 monclient(hunting): picked mon.noname-a con 0x7f67fc13f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:24.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.729+0000 7f6800882640 10 monclient(hunting): picked mon.noname-b con 0x7f67fc0a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:24.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.729+0000 7f6800882640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc0aaef0 con 0x7f67fc13f650 2024-03-29T11:15:24.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.729+0000 7f6800882640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc145bd0 con 0x7f67fc1457f0 2024-03-29T11:15:24.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.729+0000 7f6800882640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc145eb0 con 0x7f67fc0a9d70 2024-03-29T11:15:24.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.729+0000 7f6800882640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:24.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.729+0000 7f6800882640 10 monclient(hunting): authenticate will time out at 1136.926489s 2024-03-29T11:15:24.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.730+0000 7f67fa59c640 1 -- 172.21.15.157:0/639074827 learned_addr learned my addr 172.21.15.157:0/639074827 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:24.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.731+0000 7f67f959a640 1 -- 172.21.15.157:0/639074827 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307656907 0 0) 0x7f67fc0aaef0 con 0x7f67fc13f650 2024-03-29T11:15:24.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.731+0000 7f67f959a640 10 monclient(hunting): _negotiate 2024-03-29T11:15:24.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.731+0000 7f67f959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:24.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.731+0000 7f67f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:24.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.731+0000 7f67f959a640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:15:24.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.731+0000 7f67f959a640 1 -- 172.21.15.157:0/639074827 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e4003520 con 0x7f67fc13f650 2024-03-29T11:15:24.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.733+0000 7f67f959a640 1 -- 172.21.15.157:0/639074827 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 549909792 0 0) 0x7f67e4003520 con 0x7f67fc13f650 2024-03-29T11:15:24.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.733+0000 7f67f959a640 1 -- 172.21.15.157:0/639074827 >> v1:172.21.15.157:6790/0 conn(0x7f67fc0a9d70 legacy=0x7f67fc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:24.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 1 -- 172.21.15.157:0/639074827 >> v1:172.21.15.195:6789/0 conn(0x7f67fc1457f0 legacy=0x7f67fc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:24.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 10 monclient: _finish_hunting 0 2024-03-29T11:15:24.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 1 monclient: found mon.noname-a 2024-03-29T11:15:24.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:24.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:15:24.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 1 -- 172.21.15.157:0/639074827 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc146340 con 0x7f67fc13f650 2024-03-29T11:15:24.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 10 monclient: _finish_auth 0 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 10 monclient: _check_auth_tickets 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 1 -- 172.21.15.157:0/639074827 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f67f4001a10 con 0x7f67fc13f650 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f67f959a640 10 monclient: dump: 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:24.715 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:24.716 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:24.716 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:24.716 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:24.716 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:24.716 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:24.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f6800882640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:15:24.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.734+0000 7f6800882640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:24.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.737+0000 7f67f959a640 1 -- 172.21.15.157:0/639074827 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67f4001be0 con 0x7f67fc13f650 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.737+0000 7f67f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.737+0000 7f67f959a640 1 -- 172.21.15.157:0/639074827 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f67f4002030 con 0x7f67fc13f650 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.737+0000 7f67f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.737+0000 7f67f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.737+0000 7f67f959a640 10 monclient: dump: 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:24.717 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:24.718 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:24.718 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:24.718 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:24.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.738+0000 7f6800882640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:24.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.738+0000 7f6800882640 4 set_mon_vals no callback set 2024-03-29T11:15:24.719 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.741+0000 7f6800882640 10 monclient: shutdown 2024-03-29T11:15:24.719 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.741+0000 7f6800882640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:24.719 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.741+0000 7f6800882640 1 -- 172.21.15.157:0/639074827 >> v1:172.21.15.157:6789/0 conn(0x7f67fc13f650 legacy=0x7f67fc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:24.719 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.742+0000 7f6800882640 1 -- 172.21.15.157:0/639074827 shutdown_connections 2024-03-29T11:15:24.719 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.742+0000 7f6800882640 1 -- 172.21.15.157:0/639074827 >> 172.21.15.157:0/639074827 conn(0x7f67fc066a00 msgr2=0x7f67fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:24.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.743+0000 7f6800882640 1 -- 172.21.15.157:0/639074827 shutdown_connections 2024-03-29T11:15:24.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.743+0000 7f6800882640 1 -- 172.21.15.157:0/639074827 wait complete. 2024-03-29T11:15:24.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.744+0000 7f6800882640 10 monclient: build_initial_monmap 2024-03-29T11:15:24.722 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.744+0000 7f6800882640 10 monclient: monmap: 2024-03-29T11:15:24.722 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:24.722 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:24.722 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:15:24.722 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:15:24.722 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:24.722 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:24.722 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:24.723 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:15:24.723 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:15:24.723 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:24.723 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.744+0000 7f6800882640 1 librados: starting msgr at 2024-03-29T11:15:24.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.744+0000 7f6800882640 1 librados: starting objecter 2024-03-29T11:15:24.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.745+0000 7f6800882640 1 Processor -- start 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.745+0000 7f6800882640 1 -- start start 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.745+0000 7f6800882640 1 librados: setting wanted keys 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.745+0000 7f6800882640 1 librados: calling monclient init 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.745+0000 7f6800882640 10 monclient: init 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.746+0000 7f6800882640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.746+0000 7f6800882640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.747+0000 7f6800882640 10 monclient(hunting): picked mon.noname-c con 0x7f67fc0a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.747+0000 7f6800882640 10 monclient(hunting): picked mon.noname-b con 0x7f67fc13f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.747+0000 7f6800882640 10 monclient(hunting): picked mon.noname-a con 0x7f67fc1457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.747+0000 7f6800882640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc1dbb20 con 0x7f67fc1457f0 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.747+0000 7f6800882640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc06e3c0 con 0x7f67fc13f650 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.747+0000 7f6800882640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc140f00 con 0x7f67fc0a9d70 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.747+0000 7f6800882640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:24.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.747+0000 7f6800882640 10 monclient(hunting): authenticate will time out at 1136.944748s 2024-03-29T11:15:24.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.749+0000 7f67fa59c640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f67fc0a9d70 0x7f67fc1dbfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:53577/0 (socket says 172.21.15.157:53577) 2024-03-29T11:15:24.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.749+0000 7f67fa59c640 1 -- 172.21.15.157:0/3734905562 learned_addr learned my addr 172.21.15.157:0/3734905562 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:24.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.749+0000 7f67eaffd640 10 client.?.objecter ms_handle_connect 0x7f67fc0a9d70 2024-03-29T11:15:24.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.749+0000 7f67eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:24.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.749+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155091035 0 0) 0x7f67fc140f00 con 0x7f67fc0a9d70 2024-03-29T11:15:24.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.749+0000 7f67eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:24.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.749+0000 7f67eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:24.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.749+0000 7f67eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:24.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.749+0000 7f67eaffd640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.749+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67dc004eb0 con 0x7f67fc0a9d70 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.750+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1164736007 0 0) 0x7f67dc004eb0 con 0x7f67fc0a9d70 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.750+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67fc140f00 con 0x7f67fc0a9d70 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.750+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f67f0001890 con 0x7f67fc0a9d70 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.750+0000 7f67eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.750+0000 7f67eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.750+0000 7f67eaffd640 10 monclient(hunting): dump: 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:24.731 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:24.732 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:24.732 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:24.732 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:24.732 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:24.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.750+0000 7f67eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:24.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.750+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1775683886 0 0) 0x7f67fc140f00 con 0x7f67fc0a9d70 2024-03-29T11:15:24.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.750+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 >> v1:172.21.15.195:6789/0 conn(0x7f67fc13f650 legacy=0x7f67fc141a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:24.734 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.753+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 >> v1:172.21.15.157:6789/0 conn(0x7f67fc1457f0 legacy=0x7f67fc1e63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:24.734 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f67eaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:15:24.734 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f67eaffd640 1 monclient: found mon.c 2024-03-29T11:15:24.734 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f67eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:24.734 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f67eaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc1dc6c0 con 0x7f67fc0a9d70 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f67eaffd640 10 monclient: _finish_auth 0 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f67eaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f67eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f67eaffd640 10 client.?.objecter ms_handle_connect 0x7f67fc13f650 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f67eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f6800882640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f6800882640 10 monclient: _renew_subs 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f6800882640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f6800882640 1 -- 172.21.15.157:0/3734905562 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67fc142120 con 0x7f67fc0a9d70 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f6800882640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.754+0000 7f6800882640 10 monclient: _renew_subs 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.755+0000 7f6800882640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.755+0000 7f6800882640 1 -- 172.21.15.157:0/3734905562 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67fc13a6c0 con 0x7f67fc0a9d70 2024-03-29T11:15:24.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.755+0000 7f6800882640 1 librados: init done 2024-03-29T11:15:24.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.757+0000 7f67e8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:24.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.757+0000 7f67e8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:24.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.757+0000 7f67e8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:24.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.757+0000 7f67e8ff9640 1 -- 172.21.15.157:0/3734905562 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67fc075f10 con 0x7f67fc0a9d70 2024-03-29T11:15:24.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.758+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67f00047e0 con 0x7f67fc0a9d70 2024-03-29T11:15:24.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.758+0000 7f67eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:24.737 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.758+0000 7f67f8d99640 4 set_mon_vals no callback set 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.758+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f67f0001d80 con 0x7f67fc0a9d70 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.758+0000 7f67eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.758+0000 7f67eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.758+0000 7f67eaffd640 10 monclient: dump: 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:24.738 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:24.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.758+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f67f0004cf0 con 0x7f67fc0a9d70 2024-03-29T11:15:24.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.759+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f67f0084a80 con 0x7f67fc0a9d70 2024-03-29T11:15:24.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.759+0000 7f67f9d9b640 10 monclient: get_auth_request con 0x7f67dc06db80 auth_method 0 2024-03-29T11:15:24.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.760+0000 7f67eaffd640 10 client.4382.objecter ms_dispatch 0x7f67fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:24.740 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.760+0000 7f67eaffd640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:24.740 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.760+0000 7f67eaffd640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:24.740 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.760+0000 7f67eaffd640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T11:15:24.740 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.760+0000 7f67eaffd640 10 client.4382.objecter ms_handle_connect 0x7f67dc06db80 2024-03-29T11:15:24.745 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.767+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67f0050500 con 0x7f67fc0a9d70 2024-03-29T11:15:24.745 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.768+0000 7f67eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:24.745 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:24.768+0000 7f67eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:25.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.096+0000 7f67e8ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:25.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.096+0000 7f67e8ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:25.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.096+0000 7f67e8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:15:25.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.096+0000 7f67e8ff9640 1 -- 172.21.15.157:0/3734905562 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67fc0b3680 con 0x7f67fc0a9d70 2024-03-29T11:15:25.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.097+0000 7f67eaffd640 1 -- 172.21.15.157:0/3734905562 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f67f0052430 con 0x7f67fc0a9d70 2024-03-29T11:15:25.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.097+0000 7f67eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:25.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.097+0000 7f67eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:25.076 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:15:25.076 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:25.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.102+0000 7f67e8ff9640 10 librados: watch_flush enter 2024-03-29T11:15:25.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.105+0000 7f67e8ff9640 10 librados: watch_flush exit 2024-03-29T11:15:25.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.105+0000 7f67e8ff9640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T11:15:25.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.105+0000 7f67e8ff9640 10 client.4382.objecter successfully canceled tick 2024-03-29T11:15:25.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.105+0000 7f67e8ff9640 1 -- 172.21.15.157:0/3734905562 >> v1:172.21.15.157:6816/1747058247 conn(0x7f67dc06db80 legacy=0x7f67dc070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:25.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.105+0000 7f67e8ff9640 10 monclient: shutdown 2024-03-29T11:15:25.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.105+0000 7f67e8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:25.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.105+0000 7f67e8ff9640 1 -- 172.21.15.157:0/3734905562 >> v1:172.21.15.157:6790/0 conn(0x7f67fc0a9d70 legacy=0x7f67fc1dbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:25.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.106+0000 7f67e8ff9640 1 -- 172.21.15.157:0/3734905562 shutdown_connections 2024-03-29T11:15:25.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.106+0000 7f67e8ff9640 1 -- 172.21.15.157:0/3734905562 >> 172.21.15.157:0/3734905562 conn(0x7f67fc066a00 msgr2=0x7f67fc0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:25.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.106+0000 7f67e8ff9640 1 -- 172.21.15.157:0/3734905562 shutdown_connections 2024-03-29T11:15:25.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.106+0000 7f67e8ff9640 1 -- 172.21.15.157:0/3734905562 wait complete. 2024-03-29T11:15:25.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:25.107+0000 7f67e8ff9640 1 librados: shutdown 2024-03-29T11:15:25.102 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:31.909 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:32.059 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.082+0000 7ffbb0e12640 10 monclient: get_monmap_and_config 2024-03-29T11:15:32.059 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.082+0000 7ffbb0e12640 10 monclient: build_initial_monmap 2024-03-29T11:15:32.059 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.082+0000 7ffbb0e12640 10 monclient: monmap: 2024-03-29T11:15:32.059 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:32.059 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:32.059 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:15:32.082158+0000 2024-03-29T11:15:32.060 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:15:32.082158+0000 2024-03-29T11:15:32.060 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:32.060 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:32.060 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:32.060 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:15:32.060 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:15:32.060 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:32.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.083+0000 7ffbb0e12640 1 Processor -- start 2024-03-29T11:15:32.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.086+0000 7ffbb0e12640 1 -- start start 2024-03-29T11:15:32.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.086+0000 7ffbb0e12640 10 monclient: init 2024-03-29T11:15:32.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.087+0000 7ffbb0e12640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:32.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.087+0000 7ffbb0e12640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:32.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.087+0000 7ffbb0e12640 10 monclient(hunting): picked mon.noname-b con 0x7ffba4006800 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:32.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.087+0000 7ffbb0e12640 10 monclient(hunting): picked mon.noname-a con 0x7ffba4006be0 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:32.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.091+0000 7ffbb0e12640 10 monclient(hunting): picked mon.noname-c con 0x7ffba4031b60 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:32.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.091+0000 7ffbb0e12640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba4029370 con 0x7ffba4006be0 2024-03-29T11:15:32.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.091+0000 7ffbb0e12640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba40295e0 con 0x7ffba4031b60 2024-03-29T11:15:32.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.091+0000 7ffbb0e12640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba4031360 con 0x7ffba4006800 2024-03-29T11:15:32.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.091+0000 7ffbb0e12640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:32.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.091+0000 7ffbb0e12640 10 monclient(hunting): authenticate will time out at 1144.287661s 2024-03-29T11:15:32.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.092+0000 7ffbab7fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ffba4006800 0x7ffba40cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:32967/0 (socket says 172.21.15.157:32967) 2024-03-29T11:15:32.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.092+0000 7ffbab7fe640 1 -- 172.21.15.157:0/3571068189 learned_addr learned my addr 172.21.15.157:0/3571068189 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:32.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.095+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641183841 0 0) 0x7ffba4031360 con 0x7ffba4006800 2024-03-29T11:15:32.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.095+0000 7ffbaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:32.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.095+0000 7ffbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:32.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.095+0000 7ffbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:32.072 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.095+0000 7ffbaa7fc640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:15:32.072 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.095+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb9c003520 con 0x7ffba4006800 2024-03-29T11:15:32.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.095+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842474650 0 0) 0x7ffba4029370 con 0x7ffba4006be0 2024-03-29T11:15:32.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.096+0000 7ffbaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:32.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.096+0000 7ffbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:32.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.096+0000 7ffbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.096+0000 7ffbaa7fc640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.096+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba4031360 con 0x7ffba4006be0 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.096+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1731615432 0 0) 0x7ffb9c003520 con 0x7ffba4006800 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 >> v1:172.21.15.195:6789/0 conn(0x7ffba4031b60 legacy=0x7ffba40d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 >> v1:172.21.15.157:6789/0 conn(0x7ffba4006be0 legacy=0x7ffba40c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 1 monclient: found mon.noname-b 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba4031640 con 0x7ffba4006800 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:32.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ffba0001690 con 0x7ffba4006800 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbaa7fc640 10 monclient: dump: 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:32.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbb0e12640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:15:32.078 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.097+0000 7ffbb0e12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:32.078 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.099+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffba00017f0 con 0x7ffba4006800 2024-03-29T11:15:32.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.099+0000 7ffbaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:32.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.099+0000 7ffbaa7fc640 1 -- 172.21.15.157:0/3571068189 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ffba0001c00 con 0x7ffba4006800 2024-03-29T11:15:32.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.099+0000 7ffbaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:32.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.099+0000 7ffbaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:32.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.099+0000 7ffbaa7fc640 10 monclient: dump: 2024-03-29T11:15:32.079 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:32.079 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:32.079 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.099+0000 7ffbb0e12640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.099+0000 7ffbb0e12640 4 set_mon_vals no callback set 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.100+0000 7ffbb0e12640 10 monclient: shutdown 2024-03-29T11:15:32.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.100+0000 7ffbb0e12640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:32.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.100+0000 7ffbb0e12640 1 -- 172.21.15.157:0/3571068189 >> v1:172.21.15.157:6790/0 conn(0x7ffba4006800 legacy=0x7ffba40cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:32.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.101+0000 7ffbb0e12640 1 -- 172.21.15.157:0/3571068189 shutdown_connections 2024-03-29T11:15:32.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.101+0000 7ffbb0e12640 1 -- 172.21.15.157:0/3571068189 >> 172.21.15.157:0/3571068189 conn(0x7ffba40058f0 msgr2=0x7ffba4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:32.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.104+0000 7ffbb0e12640 1 -- 172.21.15.157:0/3571068189 shutdown_connections 2024-03-29T11:15:32.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.104+0000 7ffbb0e12640 1 -- 172.21.15.157:0/3571068189 wait complete. 2024-03-29T11:15:32.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.105+0000 7ffbb0e12640 10 monclient: build_initial_monmap 2024-03-29T11:15:32.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.105+0000 7ffbb0e12640 10 monclient: monmap: 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.105+0000 7ffbb0e12640 1 librados: starting msgr at 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.106+0000 7ffbb0e12640 1 librados: starting objecter 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.106+0000 7ffbb0e12640 1 Processor -- start 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.106+0000 7ffbb0e12640 1 -- start start 2024-03-29T11:15:32.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.106+0000 7ffbb0e12640 1 librados: setting wanted keys 2024-03-29T11:15:32.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.106+0000 7ffbb0e12640 1 librados: calling monclient init 2024-03-29T11:15:32.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.106+0000 7ffbb0e12640 10 monclient: init 2024-03-29T11:15:32.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.107+0000 7ffbb0e12640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:32.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.107+0000 7ffbb0e12640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:32.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.110+0000 7ffbb0e12640 10 monclient(hunting): picked mon.noname-a con 0x7ffba4006800 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:32.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.110+0000 7ffbb0e12640 10 monclient(hunting): picked mon.noname-b con 0x7ffba4031b60 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:32.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.110+0000 7ffbb0e12640 10 monclient(hunting): picked mon.noname-c con 0x7ffba402f820 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:32.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.110+0000 7ffbb0e12640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba40c7c60 con 0x7ffba4006800 2024-03-29T11:15:32.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.110+0000 7ffbb0e12640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba40c7eb0 con 0x7ffba4031b60 2024-03-29T11:15:32.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.110+0000 7ffbb0e12640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba40c8190 con 0x7ffba402f820 2024-03-29T11:15:32.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.110+0000 7ffbb0e12640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:32.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.110+0000 7ffbb0e12640 10 monclient(hunting): authenticate will time out at 1144.306361s 2024-03-29T11:15:32.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.114+0000 7ffbabfff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ffba402f820 0x7ffba415bbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:33107/0 (socket says 172.21.15.157:33107) 2024-03-29T11:15:32.091 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.114+0000 7ffbabfff640 1 -- 172.21.15.157:0/1041392633 learned_addr learned my addr 172.21.15.157:0/1041392633 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 client.?.objecter ms_handle_connect 0x7ffba4006800 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 client.?.objecter ms_handle_connect 0x7ffba402f820 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308183655 0 0) 0x7ffba40c8190 con 0x7ffba402f820 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb90004eb0 con 0x7ffba402f820 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 client.?.objecter ms_handle_connect 0x7ffba4031b60 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:32.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041878543 0 0) 0x7ffba40c7c60 con 0x7ffba4006800 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.117+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb900051d0 con 0x7ffba4006800 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2273530843 0 0) 0x7ffb90004eb0 con 0x7ffba402f820 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb90005640 con 0x7ffba402f820 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ffb98001870 con 0x7ffba402f820 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient(hunting): dump: 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:32.097 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 803444082 0 0) 0x7ffb900051d0 con 0x7ffba4006800 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb90005ce0 con 0x7ffba4006800 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ffba0001b00 con 0x7ffba4006800 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:32.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient(hunting): dump: 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2979967998 0 0) 0x7ffb90005ce0 con 0x7ffba4006800 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 >> v1:172.21.15.157:6790/0 conn(0x7ffba402f820 legacy=0x7ffba415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 >> v1:172.21.15.195:6789/0 conn(0x7ffba4031b60 legacy=0x7ffba402f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:32.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 monclient: found mon.a 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba40c72c0 con 0x7ffba4006800 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient: _finish_auth 0 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffb97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffbb0e12640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffbb0e12640 10 monclient: _renew_subs 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffbb0e12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffbb0e12640 1 -- 172.21.15.157:0/1041392633 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba415c2c0 con 0x7ffba4006800 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffbb0e12640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffbb0e12640 10 monclient: _renew_subs 2024-03-29T11:15:32.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffbb0e12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:32.101 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffbb0e12640 1 -- 172.21.15.157:0/1041392633 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba415c7f0 con 0x7ffba4006800 2024-03-29T11:15:32.101 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.118+0000 7ffbb0e12640 1 librados: init done 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.119+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffba00010e0 con 0x7ffba4006800 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.119+0000 7ffb97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.119+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ffba0001b00 con 0x7ffba4006800 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.119+0000 7ffb97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.119+0000 7ffb97fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.119+0000 7ffb97fff640 10 monclient: dump: 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:32.103 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.122+0000 7ffba97fa640 4 set_mon_vals no callback set 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.123+0000 7ffb95ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.124+0000 7ffb95ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.124+0000 7ffb95ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.124+0000 7ffb95ffb640 1 -- 172.21.15.157:0/1041392633 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb84005080 con 0x7ffba4006800 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.124+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7ffba001a900 con 0x7ffba4006800 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.125+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7ffba00543f0 con 0x7ffba4006800 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.125+0000 7ffb97fff640 10 client.4377.objecter ms_dispatch 0x7ffbac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.125+0000 7ffb97fff640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:32.104 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.125+0000 7ffb97fff640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:32.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.125+0000 7ffb97fff640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-29T11:15:32.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.125+0000 7ffbaaffd640 10 monclient: get_auth_request con 0x7ffb9006ea40 auth_method 0 2024-03-29T11:15:32.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.128+0000 7ffb97fff640 10 client.4377.objecter ms_handle_connect 0x7ffb9006ea40 2024-03-29T11:15:32.113 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.136+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffba0050690 con 0x7ffba4006800 2024-03-29T11:15:32.113 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.136+0000 7ffb97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:32.113 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.136+0000 7ffb97fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:32.432 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:15:32.433 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:32.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.454+0000 7ffb95ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:32.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.454+0000 7ffb95ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:32.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.454+0000 7ffb95ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:32.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.454+0000 7ffb95ffb640 1 -- 172.21.15.157:0/1041392633 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb84005330 con 0x7ffba4006800 2024-03-29T11:15:32.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.454+0000 7ffb97fff640 1 -- 172.21.15.157:0/1041392633 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7ffba0050830 con 0x7ffba4006800 2024-03-29T11:15:32.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.455+0000 7ffb97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:32.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.455+0000 7ffb97fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:32.437 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.458+0000 7ffb95ffb640 10 librados: watch_flush enter 2024-03-29T11:15:32.437 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.458+0000 7ffb95ffb640 10 librados: watch_flush exit 2024-03-29T11:15:32.437 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.458+0000 7ffb95ffb640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-29T11:15:32.437 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.458+0000 7ffb95ffb640 10 client.4377.objecter successfully canceled tick 2024-03-29T11:15:32.437 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.458+0000 7ffb95ffb640 1 -- 172.21.15.157:0/1041392633 >> v1:172.21.15.157:6816/1747058247 conn(0x7ffb9006ea40 legacy=0x7ffb90070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:32.438 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.458+0000 7ffb95ffb640 10 monclient: shutdown 2024-03-29T11:15:32.438 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.458+0000 7ffb95ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:32.438 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.458+0000 7ffb95ffb640 1 -- 172.21.15.157:0/1041392633 >> v1:172.21.15.157:6789/0 conn(0x7ffba4006800 legacy=0x7ffba40c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:32.438 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.458+0000 7ffb95ffb640 1 -- 172.21.15.157:0/1041392633 shutdown_connections 2024-03-29T11:15:32.438 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.458+0000 7ffb95ffb640 1 -- 172.21.15.157:0/1041392633 >> 172.21.15.157:0/1041392633 conn(0x7ffba40058f0 msgr2=0x7ffba402a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:32.438 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.459+0000 7ffb95ffb640 1 -- 172.21.15.157:0/1041392633 shutdown_connections 2024-03-29T11:15:32.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.463+0000 7ffb95ffb640 1 -- 172.21.15.157:0/1041392633 wait complete. 2024-03-29T11:15:32.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:32.463+0000 7ffb95ffb640 1 librados: shutdown 2024-03-29T11:15:32.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:39.264 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:39.421 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.443+0000 7f505e3a3640 10 monclient: get_monmap_and_config 2024-03-29T11:15:39.421 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.443+0000 7f505e3a3640 10 monclient: build_initial_monmap 2024-03-29T11:15:39.421 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.443+0000 7f505e3a3640 10 monclient: monmap: 2024-03-29T11:15:39.421 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:15:39.444179+0000 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:15:39.444179+0000 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.443+0000 7f505e3a3640 1 Processor -- start 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.444+0000 7f505e3a3640 1 -- start start 2024-03-29T11:15:39.422 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.444+0000 7f505e3a3640 10 monclient: init 2024-03-29T11:15:39.424 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.445+0000 7f505e3a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:39.424 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.445+0000 7f505e3a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:39.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.445+0000 7f505e3a3640 10 monclient(hunting): picked mon.noname-c con 0x7f5058071f40 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:39.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.445+0000 7f505e3a3640 10 monclient(hunting): picked mon.noname-a con 0x7f50580adcb0 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:39.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.445+0000 7f505e3a3640 10 monclient(hunting): picked mon.noname-b con 0x7f505813f910 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:39.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.445+0000 7f505e3a3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50580af000 con 0x7f50580adcb0 2024-03-29T11:15:39.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.445+0000 7f505e3a3640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50580ab060 con 0x7f5058071f40 2024-03-29T11:15:39.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.445+0000 7f505e3a3640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50580aec40 con 0x7f505813f910 2024-03-29T11:15:39.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.445+0000 7f505e3a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:39.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.445+0000 7f505e3a3640 10 monclient(hunting): authenticate will time out at 1151.642032s 2024-03-29T11:15:39.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.448+0000 7f505dba2640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f505813f910 0x7f5058145c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:48059/0 (socket says 172.21.15.157:48059) 2024-03-29T11:15:39.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.448+0000 7f505dba2640 1 -- 172.21.15.157:0/2019061882 learned_addr learned my addr 172.21.15.157:0/2019061882 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:39.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.449+0000 7f504ffff640 1 -- 172.21.15.157:0/2019061882 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147307850 0 0) 0x7f50580aec40 con 0x7f505813f910 2024-03-29T11:15:39.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.449+0000 7f504ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:39.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.449+0000 7f504ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:39.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.449+0000 7f504ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:39.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.449+0000 7f504ffff640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:15:39.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.449+0000 7f504ffff640 1 -- 172.21.15.157:0/2019061882 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5044003520 con 0x7f505813f910 2024-03-29T11:15:39.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.449+0000 7f504ffff640 1 -- 172.21.15.157:0/2019061882 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1792413823 0 0) 0x7f5044003520 con 0x7f505813f910 2024-03-29T11:15:39.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.449+0000 7f504ffff640 1 -- 172.21.15.157:0/2019061882 >> v1:172.21.15.195:6789/0 conn(0x7f5058071f40 legacy=0x7f505813f200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:39.431 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 1 -- 172.21.15.157:0/2019061882 >> v1:172.21.15.157:6789/0 conn(0x7f50580adcb0 legacy=0x7f5058145560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:39.431 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:39.431 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 1 monclient: found mon.noname-b 2024-03-29T11:15:39.433 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 1 -- 172.21.15.157:0/2019061882 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50580b0030 con 0x7f505813f910 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: _finish_auth 0 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 1 -- 172.21.15.157:0/2019061882 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5050001a10 con 0x7f505813f910 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: dump: 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:39.434 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f505e3a3640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f505e3a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 1 -- 172.21.15.157:0/2019061882 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5050001be0 con 0x7f505813f910 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f505e3a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f505e3a3640 4 set_mon_vals no callback set 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 1 -- 172.21.15.157:0/2019061882 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5050001ff0 con 0x7f505813f910 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:39.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.453+0000 7f504ffff640 10 monclient: dump: 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:39.436 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:39.438 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.454+0000 7f505e3a3640 10 monclient: shutdown 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.454+0000 7f505e3a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.454+0000 7f505e3a3640 1 -- 172.21.15.157:0/2019061882 >> v1:172.21.15.157:6790/0 conn(0x7f505813f910 legacy=0x7f5058145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.459+0000 7f505e3a3640 1 -- 172.21.15.157:0/2019061882 shutdown_connections 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.459+0000 7f505e3a3640 1 -- 172.21.15.157:0/2019061882 >> 172.21.15.157:0/2019061882 conn(0x7f50580673e0 msgr2=0x7f50580b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.459+0000 7f505e3a3640 1 -- 172.21.15.157:0/2019061882 shutdown_connections 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.460+0000 7f505e3a3640 1 -- 172.21.15.157:0/2019061882 wait complete. 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.460+0000 7f505e3a3640 10 monclient: build_initial_monmap 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.460+0000 7f505e3a3640 10 monclient: monmap: 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:15:39.439 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:39.440 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:39.440 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:39.440 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:15:39.440 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:15:39.440 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:39.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.461+0000 7f505e3a3640 1 librados: starting msgr at 2024-03-29T11:15:39.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.461+0000 7f505e3a3640 1 librados: starting objecter 2024-03-29T11:15:39.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.461+0000 7f505e3a3640 1 Processor -- start 2024-03-29T11:15:39.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.461+0000 7f505e3a3640 1 -- start start 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.461+0000 7f505e3a3640 1 librados: setting wanted keys 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.461+0000 7f505e3a3640 1 librados: calling monclient init 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.461+0000 7f505e3a3640 10 monclient: init 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.462+0000 7f505e3a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.462+0000 7f505e3a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.462+0000 7f505e3a3640 10 monclient(hunting): picked mon.noname-a con 0x7f5058071f40 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.462+0000 7f505e3a3640 10 monclient(hunting): picked mon.noname-b con 0x7f50580adcb0 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.462+0000 7f505e3a3640 10 monclient(hunting): picked mon.noname-c con 0x7f505813f910 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.462+0000 7f505e3a3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5058134820 con 0x7f5058071f40 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.462+0000 7f505e3a3640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5058134b00 con 0x7f50580adcb0 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.462+0000 7f505e3a3640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5058132430 con 0x7f505813f910 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.462+0000 7f505e3a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:39.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.462+0000 7f505e3a3640 10 monclient(hunting): authenticate will time out at 1151.659441s 2024-03-29T11:15:39.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.465+0000 7f505cba0640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f50580adcb0 0x7f50581daa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:50569/0 (socket says 172.21.15.157:50569) 2024-03-29T11:15:39.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.465+0000 7f505cba0640 1 -- 172.21.15.157:0/792656649 learned_addr learned my addr 172.21.15.157:0/792656649 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:39.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.465+0000 7f504dffb640 10 client.?.objecter ms_handle_connect 0x7f50580adcb0 2024-03-29T11:15:39.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.465+0000 7f504dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:39.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.465+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557366923 0 0) 0x7f5058134b00 con 0x7f50580adcb0 2024-03-29T11:15:39.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.465+0000 7f504dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:39.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.465+0000 7f504dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:39.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.465+0000 7f504dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:39.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.465+0000 7f504dffb640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:15:39.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.465+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c004eb0 con 0x7f50580adcb0 2024-03-29T11:15:39.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.466+0000 7f504dffb640 10 client.?.objecter ms_handle_connect 0x7f5058071f40 2024-03-29T11:15:39.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.466+0000 7f504dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:39.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.466+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779435074 0 0) 0x7f5058134820 con 0x7f5058071f40 2024-03-29T11:15:39.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.466+0000 7f504dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:39.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.466+0000 7f504dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:39.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.466+0000 7f504dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:39.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.466+0000 7f504dffb640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:15:39.444 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.466+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5058134b00 con 0x7f5058071f40 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.467+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1236203176 0 0) 0x7f5058134b00 con 0x7f5058071f40 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.467+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5058134820 con 0x7f5058071f40 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.467+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5054001790 con 0x7f5058071f40 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.467+0000 7f504dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.467+0000 7f504dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.467+0000 7f504dffb640 10 monclient(hunting): dump: 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:39.445 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:39.446 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:39.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.467+0000 7f504dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:39.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.467+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3567272956 0 0) 0x7f5058134820 con 0x7f5058071f40 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.467+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 >> v1:172.21.15.157:6790/0 conn(0x7f505813f910 legacy=0x7f50581e1c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 >> v1:172.21.15.195:6789/0 conn(0x7f50580adcb0 legacy=0x7f50581daa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f504dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f504dffb640 1 monclient: found mon.a 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f504dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f504dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50581335b0 con 0x7f5058071f40 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f504dffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f504dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f504dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f505e3a3640 5 monclient: authenticate success, global_id 4383 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f505e3a3640 10 monclient: _renew_subs 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f505e3a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:39.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f505e3a3640 1 -- 172.21.15.157:0/792656649 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50581db120 con 0x7f5058071f40 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f505e3a3640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f505e3a3640 10 monclient: _renew_subs 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f505e3a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f505e3a3640 1 -- 172.21.15.157:0/792656649 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50581e3360 con 0x7f5058071f40 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.468+0000 7f505e3a3640 1 librados: init done 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.469+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5054001ca0 con 0x7f5058071f40 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.469+0000 7f504dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.469+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5054003790 con 0x7f5058071f40 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.469+0000 7f504dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.469+0000 7f504dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.469+0000 7f504dffb640 10 monclient: dump: 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:39.450 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.469+0000 7f504f7fe640 4 set_mon_vals no callback set 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.469+0000 7f505e3a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.470+0000 7f505e3a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.470+0000 7f505e3a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:39.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.470+0000 7f505e3a3640 1 -- 172.21.15.157:0/792656649 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5038005080 con 0x7f5058071f40 2024-03-29T11:15:39.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.471+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f5054019760 con 0x7f5058071f40 2024-03-29T11:15:39.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.472+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f50540837a0 con 0x7f5058071f40 2024-03-29T11:15:39.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.472+0000 7f504dffb640 10 client.4383.objecter ms_dispatch 0x7f5058000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:39.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.472+0000 7f504dffb640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:39.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.472+0000 7f504dffb640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:39.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.472+0000 7f504dffb640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-29T11:15:39.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.475+0000 7f505cba0640 10 monclient: get_auth_request con 0x7f503c06dce0 auth_method 0 2024-03-29T11:15:39.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.476+0000 7f504dffb640 10 client.4383.objecter ms_handle_connect 0x7f503c06dce0 2024-03-29T11:15:39.460 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.482+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f505404eeb0 con 0x7f5058071f40 2024-03-29T11:15:39.460 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.482+0000 7f504dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:39.460 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.482+0000 7f504dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:39.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.805+0000 7f505e3a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:39.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.805+0000 7f505e3a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:39.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.805+0000 7f505e3a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:15:39.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.805+0000 7f505e3a3640 1 -- 172.21.15.157:0/792656649 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5038005330 con 0x7f5058071f40 2024-03-29T11:15:39.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.806+0000 7f504dffb640 1 -- 172.21.15.157:0/792656649 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f5054050fb0 con 0x7f5058071f40 2024-03-29T11:15:39.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.806+0000 7f504dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:39.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.806+0000 7f504dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:39.785 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:15:39.785 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:39.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.810+0000 7f505e3a3640 10 librados: watch_flush enter 2024-03-29T11:15:39.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.810+0000 7f505e3a3640 10 librados: watch_flush exit 2024-03-29T11:15:39.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.810+0000 7f505e3a3640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-29T11:15:39.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.810+0000 7f505e3a3640 10 client.4383.objecter successfully canceled tick 2024-03-29T11:15:39.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.810+0000 7f505e3a3640 1 -- 172.21.15.157:0/792656649 >> v1:172.21.15.157:6816/1747058247 conn(0x7f503c06dce0 legacy=0x7f503c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:39.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.810+0000 7f505e3a3640 10 monclient: shutdown 2024-03-29T11:15:39.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.810+0000 7f505e3a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:39.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.810+0000 7f505e3a3640 1 -- 172.21.15.157:0/792656649 >> v1:172.21.15.157:6789/0 conn(0x7f5058071f40 legacy=0x7f5058132ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:39.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.810+0000 7f505e3a3640 1 -- 172.21.15.157:0/792656649 shutdown_connections 2024-03-29T11:15:39.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.810+0000 7f505e3a3640 1 -- 172.21.15.157:0/792656649 >> 172.21.15.157:0/792656649 conn(0x7f50580673e0 msgr2=0x7f5058069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:39.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.811+0000 7f505e3a3640 1 -- 172.21.15.157:0/792656649 shutdown_connections 2024-03-29T11:15:39.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.811+0000 7f505e3a3640 1 -- 172.21.15.157:0/792656649 wait complete. 2024-03-29T11:15:39.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:39.811+0000 7f505e3a3640 1 librados: shutdown 2024-03-29T11:15:39.807 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:46.614 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:46.763 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.786+0000 7ffbd7fff640 10 monclient: get_monmap_and_config 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.786+0000 7ffbd7fff640 10 monclient: build_initial_monmap 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.786+0000 7ffbd7fff640 10 monclient: monmap: 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:15:46.787130+0000 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:15:46.787130+0000 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:15:46.764 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:46.766 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.787+0000 7ffbd7fff640 1 Processor -- start 2024-03-29T11:15:46.766 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.787+0000 7ffbd7fff640 1 -- start start 2024-03-29T11:15:46.766 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.787+0000 7ffbd7fff640 10 monclient: init 2024-03-29T11:15:46.766 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.788+0000 7ffbd7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:46.766 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.788+0000 7ffbd7fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:46.766 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.788+0000 7ffbd7fff640 10 monclient(hunting): picked mon.noname-a con 0x7ffbd80b4820 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:46.766 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.788+0000 7ffbd7fff640 10 monclient(hunting): picked mon.noname-c con 0x7ffbd80a9ce0 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:46.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.788+0000 7ffbd7fff640 10 monclient(hunting): picked mon.noname-b con 0x7ffbd80b3cd0 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:46.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.788+0000 7ffbd7fff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd80b4180 con 0x7ffbd80b4820 2024-03-29T11:15:46.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.788+0000 7ffbd7fff640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd80aaf60 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.788+0000 7ffbd7fff640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd813c200 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.788+0000 7ffbd7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:46.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.788+0000 7ffbd7fff640 10 monclient(hunting): authenticate will time out at 1158.985659s 2024-03-29T11:15:46.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.793+0000 7ffbd77fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ffbd80b3cd0 0x7ffbd814d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:42017/0 (socket says 172.21.15.157:42017) 2024-03-29T11:15:46.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.793+0000 7ffbd77fe640 1 -- 172.21.15.157:0/543277414 learned_addr learned my addr 172.21.15.157:0/543277414 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:46.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549478459 0 0) 0x7ffbd813c200 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:46.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:46.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbbc003520 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285357165 0 0) 0x7ffbc4001040 con 0x7ffbd80b4820 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd813c200 con 0x7ffbd80b4820 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369507165 0 0) 0x7ffbd80aaf60 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:15:46.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.796+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc4001040 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1845105953 0 0) 0x7ffbbc003520 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 >> v1:172.21.15.195:6789/0 conn(0x7ffbd80a9ce0 legacy=0x7ffbd8145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 >> v1:172.21.15.157:6789/0 conn(0x7ffbd80b4820 legacy=0x7ffbd813baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd8145e80 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ffbd0001a10 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:46.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: dump: 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbd0001bf0 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:46.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 1 -- 172.21.15.157:0/543277414 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ffbd0001a10 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd5ffb640 10 monclient: dump: 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd7fff640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:15:46.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd7fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:46.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.797+0000 7ffbd7fff640 4 set_mon_vals no callback set 2024-03-29T11:15:46.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.798+0000 7ffbd7fff640 10 monclient: shutdown 2024-03-29T11:15:46.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.798+0000 7ffbd7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:46.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.798+0000 7ffbd7fff640 1 -- 172.21.15.157:0/543277414 >> v1:172.21.15.157:6790/0 conn(0x7ffbd80b3cd0 legacy=0x7ffbd814d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:46.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.798+0000 7ffbd7fff640 1 -- 172.21.15.157:0/543277414 shutdown_connections 2024-03-29T11:15:46.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.798+0000 7ffbd7fff640 1 -- 172.21.15.157:0/543277414 >> 172.21.15.157:0/543277414 conn(0x7ffbd806d070 msgr2=0x7ffbd8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:46.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.798+0000 7ffbd7fff640 1 -- 172.21.15.157:0/543277414 shutdown_connections 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.798+0000 7ffbd7fff640 1 -- 172.21.15.157:0/543277414 wait complete. 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.799+0000 7ffbd7fff640 10 monclient: build_initial_monmap 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.800+0000 7ffbd7fff640 10 monclient: monmap: 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:46.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.800+0000 7ffbd7fff640 1 librados: starting msgr at 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.800+0000 7ffbd7fff640 1 librados: starting objecter 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.801+0000 7ffbd7fff640 1 Processor -- start 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.801+0000 7ffbd7fff640 1 -- start start 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.801+0000 7ffbd7fff640 1 librados: setting wanted keys 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.801+0000 7ffbd7fff640 1 librados: calling monclient init 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.801+0000 7ffbd7fff640 10 monclient: init 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.802+0000 7ffbd7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.802+0000 7ffbd7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.802+0000 7ffbd7fff640 10 monclient(hunting): picked mon.noname-b con 0x7ffbd80a9ce0 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.802+0000 7ffbd7fff640 10 monclient(hunting): picked mon.noname-c con 0x7ffbd80b3cd0 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.802+0000 7ffbd7fff640 10 monclient(hunting): picked mon.noname-a con 0x7ffbd80b4820 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.802+0000 7ffbd7fff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd80706b0 con 0x7ffbd80b4820 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.802+0000 7ffbd7fff640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd80abd60 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.802+0000 7ffbd7fff640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd813b6d0 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.802+0000 7ffbd7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:46.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.802+0000 7ffbd7fff640 10 monclient(hunting): authenticate will time out at 1158.999543s 2024-03-29T11:15:46.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.806+0000 7ffbd67fc640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ffbd80b3cd0 0x7ffbd813a8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:42105/0 (socket says 172.21.15.157:42105) 2024-03-29T11:15:46.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.806+0000 7ffbd67fc640 1 -- 172.21.15.157:0/1864616992 learned_addr learned my addr 172.21.15.157:0/1864616992 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.806+0000 7ffbcf7fe640 10 client.?.objecter ms_handle_connect 0x7ffbd80b3cd0 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.806+0000 7ffbcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.806+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088258657 0 0) 0x7ffbd813b6d0 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.806+0000 7ffbcf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.806+0000 7ffbcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.806+0000 7ffbcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.806+0000 7ffbcf7fe640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.807+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc0004eb0 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.808+0000 7ffbcf7fe640 10 client.?.objecter ms_handle_connect 0x7ffbd80a9ce0 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.808+0000 7ffbcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.808+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283170301 0 0) 0x7ffbd80abd60 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.808+0000 7ffbcf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:46.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.808+0000 7ffbcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.808+0000 7ffbcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.808+0000 7ffbcf7fe640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.809+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd813b6d0 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.809+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3865293874 0 0) 0x7ffbd813b6d0 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.809+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbd80abd60 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.809+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 737088378 0 0) 0x7ffbc0004eb0 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.809+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbd813b6d0 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.809+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ffbc4001620 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient(hunting): dump: 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:46.790 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ffbc8001860 con 0x7ffbd80b3cd0 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient(hunting): dump: 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:46.791 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2492430932 0 0) 0x7ffbd80abd60 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 >> v1:172.21.15.157:6790/0 conn(0x7ffbd80b3cd0 legacy=0x7ffbd813a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 >> v1:172.21.15.157:6789/0 conn(0x7ffbd80b4820 legacy=0x7ffbd813afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:46.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 1 monclient: found mon.b 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd8070b40 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbd7fff640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbd7fff640 10 monclient: _renew_subs 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbd7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbd7fff640 1 -- 172.21.15.157:0/1864616992 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd806fba0 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbd7fff640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbd7fff640 10 monclient: _renew_subs 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbd7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:15:46.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbd7fff640 1 -- 172.21.15.157:0/1864616992 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd81d3530 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbd7fff640 1 librados: init done 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.810+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbc4001b90 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.811+0000 7ffbcf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.811+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ffbc4003610 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.811+0000 7ffbcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.811+0000 7ffbcf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.811+0000 7ffbcf7fe640 10 monclient: dump: 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:46.794 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:46.795 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:46.795 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:46.795 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.811+0000 7ffbd4ff9640 4 set_mon_vals no callback set 2024-03-29T11:15:46.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.812+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7ffbc40195e0 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.813+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7ffbc40834b0 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.813+0000 7ffbcf7fe640 10 client.4366.objecter ms_dispatch 0x7ffbd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:46.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.813+0000 7ffbcf7fe640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:46.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.813+0000 7ffbcf7fe640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:46.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.813+0000 7ffbcf7fe640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T11:15:46.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.813+0000 7ffbd67fc640 10 monclient: get_auth_request con 0x7ffbc006ded0 auth_method 0 2024-03-29T11:15:46.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.814+0000 7ffbcf7fe640 10 client.4366.objecter ms_handle_connect 0x7ffbc006ded0 2024-03-29T11:15:46.796 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.814+0000 7ffbd7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:46.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.814+0000 7ffbd7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:46.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.814+0000 7ffbd7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:15:46.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.814+0000 7ffbd7fff640 1 -- 172.21.15.157:0/1864616992 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb4005080 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.822+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffbc404ed90 con 0x7ffbd80a9ce0 2024-03-29T11:15:46.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.822+0000 7ffbcf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:46.800 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:46.822+0000 7ffbcf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:47.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.151+0000 7ffbd7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:47.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.151+0000 7ffbd7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:47.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.151+0000 7ffbd7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:15:47.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.151+0000 7ffbd7fff640 1 -- 172.21.15.157:0/1864616992 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbb4005330 con 0x7ffbd80a9ce0 2024-03-29T11:15:47.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.153+0000 7ffbcf7fe640 1 -- 172.21.15.157:0/1864616992 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7ffbc4050cc0 con 0x7ffbd80a9ce0 2024-03-29T11:15:47.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.153+0000 7ffbcf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:47.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.153+0000 7ffbcf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:47.133 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:15:47.133 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:47.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.157+0000 7ffbd7fff640 10 librados: watch_flush enter 2024-03-29T11:15:47.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.160+0000 7ffbd7fff640 10 librados: watch_flush exit 2024-03-29T11:15:47.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.160+0000 7ffbd7fff640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T11:15:47.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.160+0000 7ffbd7fff640 10 client.4366.objecter successfully canceled tick 2024-03-29T11:15:47.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.160+0000 7ffbd7fff640 1 -- 172.21.15.157:0/1864616992 >> v1:172.21.15.157:6816/1747058247 conn(0x7ffbc006ded0 legacy=0x7ffbc0070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:47.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.160+0000 7ffbd7fff640 10 monclient: shutdown 2024-03-29T11:15:47.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.160+0000 7ffbd7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:47.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.160+0000 7ffbd7fff640 1 -- 172.21.15.157:0/1864616992 >> v1:172.21.15.195:6789/0 conn(0x7ffbd80a9ce0 legacy=0x7ffbd81d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:47.139 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.161+0000 7ffbd7fff640 1 -- 172.21.15.157:0/1864616992 shutdown_connections 2024-03-29T11:15:47.139 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.161+0000 7ffbd7fff640 1 -- 172.21.15.157:0/1864616992 >> 172.21.15.157:0/1864616992 conn(0x7ffbd806d070 msgr2=0x7ffbd80b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:47.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.163+0000 7ffbd7fff640 1 -- 172.21.15.157:0/1864616992 shutdown_connections 2024-03-29T11:15:47.142 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.165+0000 7ffbd7fff640 1 -- 172.21.15.157:0/1864616992 wait complete. 2024-03-29T11:15:47.142 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:47.165+0000 7ffbd7fff640 1 librados: shutdown 2024-03-29T11:15:47.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:53.970 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:54.115 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.137+0000 7fd866bd6640 10 monclient: get_monmap_and_config 2024-03-29T11:15:54.115 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.137+0000 7fd866bd6640 10 monclient: build_initial_monmap 2024-03-29T11:15:54.115 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.137+0000 7fd866bd6640 10 monclient: monmap: 2024-03-29T11:15:54.116 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:54.116 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:54.116 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:15:54.139037+0000 2024-03-29T11:15:54.116 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:15:54.139037+0000 2024-03-29T11:15:54.116 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:54.116 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:54.116 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:54.116 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:15:54.116 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:15:54.116 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:54.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.138+0000 7fd866bd6640 1 Processor -- start 2024-03-29T11:15:54.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 1 -- start start 2024-03-29T11:15:54.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 10 monclient: init 2024-03-29T11:15:54.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 10 monclient(hunting): picked mon.noname-c con 0x7fd8600b0770 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 10 monclient(hunting): picked mon.noname-a con 0x7fd86014a700 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 10 monclient(hunting): picked mon.noname-b con 0x7fd8600ac5a0 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8600aba70 con 0x7fd86014a700 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8600aecb0 con 0x7fd8600b0770 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8600af150 con 0x7fd8600ac5a0 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.139+0000 7fd866bd6640 10 monclient(hunting): authenticate will time out at 1166.336794s 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.140+0000 7fd865173640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fd8600ac5a0 0x7fd860151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:33448/0 (socket says 172.21.15.157:33448) 2024-03-29T11:15:54.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.140+0000 7fd865173640 1 -- 172.21.15.157:0/4060277238 learned_addr learned my addr 172.21.15.157:0/4060277238 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:54.120 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.140+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596677728 0 0) 0x7fd8600aba70 con 0x7fd86014a700 2024-03-29T11:15:54.120 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.140+0000 7fd85f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:54.120 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.140+0000 7fd85f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:54.120 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.140+0000 7fd85f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.140+0000 7fd85f7fe640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.140+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd84c003520 con 0x7fd86014a700 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617334184 0 0) 0x7fd8600aecb0 con 0x7fd8600b0770 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8600aba70 con 0x7fd8600b0770 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150444669 0 0) 0x7fd8600af150 con 0x7fd8600ac5a0 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8600aecb0 con 0x7fd8600ac5a0 2024-03-29T11:15:54.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1385633644 0 0) 0x7fd84c003520 con 0x7fd86014a700 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 >> v1:172.21.15.157:6790/0 conn(0x7fd8600ac5a0 legacy=0x7fd860151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 >> v1:172.21.15.195:6789/0 conn(0x7fd8600b0770 legacy=0x7fd860149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8600ac9d0 con 0x7fd86014a700 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd858001690 con 0x7fd86014a700 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:54.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.141+0000 7fd85f7fe640 10 monclient: dump: 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.142+0000 7fd866bd6640 5 monclient: authenticate success, global_id 4389 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.142+0000 7fd866bd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.143+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8580017f0 con 0x7fd86014a700 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.143+0000 7fd85f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:54.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.143+0000 7fd866bd6640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:54.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.143+0000 7fd866bd6640 4 set_mon_vals no callback set 2024-03-29T11:15:54.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.143+0000 7fd85f7fe640 1 -- 172.21.15.157:0/4060277238 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd858001c40 con 0x7fd86014a700 2024-03-29T11:15:54.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.143+0000 7fd85f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:54.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.143+0000 7fd85f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:54.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.143+0000 7fd85f7fe640 10 monclient: dump: 2024-03-29T11:15:54.129 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.144+0000 7fd866bd6640 10 monclient: shutdown 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.144+0000 7fd866bd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.144+0000 7fd866bd6640 1 -- 172.21.15.157:0/4060277238 >> v1:172.21.15.157:6789/0 conn(0x7fd86014a700 legacy=0x7fd8600abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.144+0000 7fd866bd6640 1 -- 172.21.15.157:0/4060277238 shutdown_connections 2024-03-29T11:15:54.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.144+0000 7fd866bd6640 1 -- 172.21.15.157:0/4060277238 >> 172.21.15.157:0/4060277238 conn(0x7fd86006b660 msgr2=0x7fd86006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.144+0000 7fd866bd6640 1 -- 172.21.15.157:0/4060277238 shutdown_connections 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.144+0000 7fd866bd6640 1 -- 172.21.15.157:0/4060277238 wait complete. 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.145+0000 7fd866bd6640 10 monclient: build_initial_monmap 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.145+0000 7fd866bd6640 10 monclient: monmap: 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:15:54.131 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.145+0000 7fd866bd6640 1 librados: starting msgr at 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.145+0000 7fd866bd6640 1 librados: starting objecter 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.145+0000 7fd866bd6640 1 Processor -- start 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.146+0000 7fd866bd6640 1 -- start start 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.146+0000 7fd866bd6640 1 librados: setting wanted keys 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.146+0000 7fd866bd6640 1 librados: calling monclient init 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.146+0000 7fd866bd6640 10 monclient: init 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.147+0000 7fd866bd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.147+0000 7fd866bd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.147+0000 7fd866bd6640 10 monclient(hunting): picked mon.noname-a con 0x7fd8600ac5a0 addr v1:172.21.15.157:6789/0 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.147+0000 7fd866bd6640 10 monclient(hunting): picked mon.noname-c con 0x7fd8600b0770 addr v1:172.21.15.157:6790/0 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.147+0000 7fd866bd6640 10 monclient(hunting): picked mon.noname-b con 0x7fd860125820 addr v1:172.21.15.195:6789/0 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.147+0000 7fd866bd6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8601271c0 con 0x7fd8600ac5a0 2024-03-29T11:15:54.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.147+0000 7fd866bd6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd860124e40 con 0x7fd860125820 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.147+0000 7fd864972640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd8600ac5a0 0x7fd8601db530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:39966/0 (socket says 172.21.15.157:39966) 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.147+0000 7fd864972640 1 -- 172.21.15.157:0/614330168 learned_addr learned my addr 172.21.15.157:0/614330168 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd860125cb0 con 0x7fd8600b0770 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd866bd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd866bd6640 10 monclient(hunting): authenticate will time out at 1166.345040s 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 client.?.objecter ms_handle_connect 0x7fd8600ac5a0 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 client.?.objecter ms_handle_connect 0x7fd860125820 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167003794 0 0) 0x7fd860124e40 con 0x7fd860125820 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:15:54.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd840004eb0 con 0x7fd860125820 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298513394 0 0) 0x7fd8601271c0 con 0x7fd8600ac5a0 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.148+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd860124e40 con 0x7fd8600ac5a0 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 client.?.objecter ms_handle_connect 0x7fd8600b0770 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115578775 0 0) 0x7fd860125cb0 con 0x7fd8600b0770 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8601271c0 con 0x7fd8600b0770 2024-03-29T11:15:54.134 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2769714855 0 0) 0x7fd840004eb0 con 0x7fd860125820 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd860125cb0 con 0x7fd860125820 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd848001870 con 0x7fd860125820 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): dump: 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:54.135 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2325976618 0 0) 0x7fd8601271c0 con 0x7fd8600b0770 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd840004eb0 con 0x7fd8600b0770 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd8580014b0 con 0x7fd8600b0770 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 10 monclient(hunting): dump: 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:54.136 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.149+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2940841300 0 0) 0x7fd860125cb0 con 0x7fd860125820 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 >> v1:172.21.15.157:6790/0 conn(0x7fd8600b0770 legacy=0x7fd860143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 >> v1:172.21.15.157:6789/0 conn(0x7fd8600ac5a0 legacy=0x7fd8601db530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 1 monclient: found mon.b 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860144ac0 con 0x7fd860125820 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd866bd6640 5 monclient: authenticate success, global_id 4372 2024-03-29T11:15:54.137 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd866bd6640 10 monclient: _renew_subs 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd866bd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8601dbc40 con 0x7fd860125820 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd866bd6640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd866bd6640 10 monclient: _renew_subs 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd866bd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8601263b0 con 0x7fd860125820 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd866bd6640 1 librados: init done 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd848001d60 con 0x7fd860125820 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.150+0000 7fd85effd640 4 set_mon_vals no callback set 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.151+0000 7fd866bd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.151+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd848004980 con 0x7fd860125820 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.151+0000 7fd85d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:54.138 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.151+0000 7fd85d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.151+0000 7fd85d7fa640 10 monclient: dump: 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.151+0000 7fd866bd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.151+0000 7fd866bd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:15:54.139 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.151+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd844005080 con 0x7fd860125820 2024-03-29T11:15:54.140 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.153+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fd84801a950 con 0x7fd860125820 2024-03-29T11:15:54.140 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.154+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fd848084930 con 0x7fd860125820 2024-03-29T11:15:54.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.154+0000 7fd85d7fa640 10 client.4372.objecter ms_dispatch 0x7fd860000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:54.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.154+0000 7fd85d7fa640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:54.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.154+0000 7fd85d7fa640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:54.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.154+0000 7fd85d7fa640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-29T11:15:54.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.154+0000 7fd864972640 10 monclient: get_auth_request con 0x7fd84006e0b0 auth_method 0 2024-03-29T11:15:54.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.154+0000 7fd85d7fa640 10 client.4372.objecter ms_handle_connect 0x7fd84006e0b0 2024-03-29T11:15:54.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.159+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd848050040 con 0x7fd860125820 2024-03-29T11:15:54.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.159+0000 7fd85d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:54.141 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.159+0000 7fd85d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:54.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.541+0000 7fd866bd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:54.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.541+0000 7fd866bd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:54.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.541+0000 7fd866bd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:15:54.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.541+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd844005330 con 0x7fd860125820 2024-03-29T11:15:54.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.542+0000 7fd85d7fa640 1 -- 172.21.15.157:0/614330168 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fd848052140 con 0x7fd860125820 2024-03-29T11:15:54.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.542+0000 7fd85d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:54.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.542+0000 7fd85d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:54.520 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:15:54.520 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:54.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.545+0000 7fd866bd6640 10 librados: watch_flush enter 2024-03-29T11:15:54.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.545+0000 7fd866bd6640 10 librados: watch_flush exit 2024-03-29T11:15:54.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.545+0000 7fd866bd6640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-29T11:15:54.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.545+0000 7fd866bd6640 10 client.4372.objecter successfully canceled tick 2024-03-29T11:15:54.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.545+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 >> v1:172.21.15.157:6816/1747058247 conn(0x7fd84006e0b0 legacy=0x7fd840070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:54.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.545+0000 7fd866bd6640 10 monclient: shutdown 2024-03-29T11:15:54.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.545+0000 7fd866bd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:54.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.545+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 >> v1:172.21.15.195:6789/0 conn(0x7fd860125820 legacy=0x7fd8601e6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:54.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.546+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 shutdown_connections 2024-03-29T11:15:54.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.546+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 >> 172.21.15.157:0/614330168 conn(0x7fd86006b660 msgr2=0x7fd860149530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:54.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.546+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 shutdown_connections 2024-03-29T11:15:54.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.548+0000 7fd866bd6640 1 -- 172.21.15.157:0/614330168 wait complete. 2024-03-29T11:15:54.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:15:54.548+0000 7fd866bd6640 1 librados: shutdown 2024-03-29T11:15:54.545 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:01.352 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:01.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.524+0000 7f440776b640 10 monclient: get_monmap_and_config 2024-03-29T11:16:01.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.524+0000 7f440776b640 10 monclient: build_initial_monmap 2024-03-29T11:16:01.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.524+0000 7f440776b640 10 monclient: monmap: 2024-03-29T11:16:01.502 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:01.502 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:01.502 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:16:01.525350+0000 2024-03-29T11:16:01.502 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:16:01.525350+0000 2024-03-29T11:16:01.502 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:01.502 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:01.502 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:01.502 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:16:01.503 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:16:01.503 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:01.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.525+0000 7f440776b640 1 Processor -- start 2024-03-29T11:16:01.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.525+0000 7f440776b640 1 -- start start 2024-03-29T11:16:01.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.525+0000 7f440776b640 10 monclient: init 2024-03-29T11:16:01.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f440776b640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:01.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f440776b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:01.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f440776b640 10 monclient(hunting): picked mon.noname-a con 0x7f44001457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:01.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f440776b640 10 monclient(hunting): picked mon.noname-c con 0x7f440013f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:01.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f440776b640 10 monclient(hunting): picked mon.noname-b con 0x7f44000a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:01.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f440776b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44000aaef0 con 0x7f44001457f0 2024-03-29T11:16:01.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f440776b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4400145bd0 con 0x7f440013f650 2024-03-29T11:16:01.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f440776b640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4400145eb0 con 0x7f44000a9d70 2024-03-29T11:16:01.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f440776b640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:01.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f440776b640 10 monclient(hunting): authenticate will time out at 1173.723543s 2024-03-29T11:16:01.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f4405d08640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f44000a9d70 0x7f440014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:51478/0 (socket says 172.21.15.157:51478) 2024-03-29T11:16:01.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f4405d08640 1 -- 172.21.15.157:0/1926424362 learned_addr learned my addr 172.21.15.157:0/1926424362 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.526+0000 7f4404d06640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f440013f650 0x7f44000b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:37698/0 (socket says 172.21.15.157:37698) 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348585674 0 0) 0x7f44000aaef0 con 0x7f44001457f0 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43ec003520 con 0x7f44001457f0 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618097169 0 0) 0x7f4400145eb0 con 0x7f44000a9d70 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:01.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43ec003840 con 0x7f44000a9d70 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3688659801 0 0) 0x7f43ec003520 con 0x7f44001457f0 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 >> v1:172.21.15.157:6790/0 conn(0x7f44000a9d70 legacy=0x7f440014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 >> v1:172.21.15.195:6789/0 conn(0x7f440013f650 legacy=0x7f44000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 1 monclient: found mon.noname-a 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400146340 con 0x7f44001457f0 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient: _finish_auth 0 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:01.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f440776b640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f440776b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f43f80015a0 con 0x7f44001457f0 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f43f7fff640 10 monclient: dump: 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:01.510 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.527+0000 7f440776b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43f8001780 con 0x7f44001457f0 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f43f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f440776b640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f440776b640 4 set_mon_vals no callback set 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f43f7fff640 1 -- 172.21.15.157:0/1926424362 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f43f8001b90 con 0x7f44001457f0 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f43f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f43f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f43f7fff640 10 monclient: dump: 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:01.511 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f440776b640 10 monclient: shutdown 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f440776b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f440776b640 1 -- 172.21.15.157:0/1926424362 >> v1:172.21.15.157:6789/0 conn(0x7f44001457f0 legacy=0x7f440013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f440776b640 1 -- 172.21.15.157:0/1926424362 shutdown_connections 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.528+0000 7f440776b640 1 -- 172.21.15.157:0/1926424362 >> 172.21.15.157:0/1926424362 conn(0x7f4400066a00 msgr2=0x7f4400066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.529+0000 7f440776b640 1 -- 172.21.15.157:0/1926424362 shutdown_connections 2024-03-29T11:16:01.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.529+0000 7f440776b640 1 -- 172.21.15.157:0/1926424362 wait complete. 2024-03-29T11:16:01.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.530+0000 7f440776b640 10 monclient: build_initial_monmap 2024-03-29T11:16:01.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.530+0000 7f440776b640 10 monclient: monmap: 2024-03-29T11:16:01.516 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:01.516 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:01.516 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:16:01.516 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:16:01.516 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:01.516 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:01.516 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.530+0000 7f440776b640 1 librados: starting msgr at 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.530+0000 7f440776b640 1 librados: starting objecter 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.530+0000 7f440776b640 1 Processor -- start 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.531+0000 7f440776b640 1 -- start start 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.531+0000 7f440776b640 1 librados: setting wanted keys 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.531+0000 7f440776b640 1 librados: calling monclient init 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.531+0000 7f440776b640 10 monclient: init 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f440776b640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f440776b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f440776b640 10 monclient(hunting): picked mon.noname-b con 0x7f44000a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f440776b640 10 monclient(hunting): picked mon.noname-a con 0x7f440013f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f440776b640 10 monclient(hunting): picked mon.noname-c con 0x7f4400071c00 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:01.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f440776b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44001427b0 con 0x7f440013f650 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f440776b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4400142cd0 con 0x7f44000a9d70 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f4405d08640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f4400071c00 0x7f44001d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:51662/0 (socket says 172.21.15.157:51662) 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f4405d08640 1 -- 172.21.15.157:0/3149403650 learned_addr learned my addr 172.21.15.157:0/3149403650 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4400144940 con 0x7f4400071c00 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f440776b640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f440776b640 10 monclient(hunting): authenticate will time out at 1173.729313s 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f43f5ffb640 10 client.?.objecter ms_handle_connect 0x7f4400071c00 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f43f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f43f5ffb640 10 client.?.objecter ms_handle_connect 0x7f440013f650 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f43f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279302583 0 0) 0x7f4400144940 con 0x7f4400071c00 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f43f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:01.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f43f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f43f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f43f5ffb640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.532+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e4004eb0 con 0x7f4400071c00 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 client.?.objecter ms_handle_connect 0x7f44000a9d70 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731313230 0 0) 0x7f44001427b0 con 0x7f440013f650 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4400144940 con 0x7f440013f650 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3320897208 0 0) 0x7f43e4004eb0 con 0x7f4400071c00 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44001427b0 con 0x7f4400071c00 2024-03-29T11:16:01.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1967181937 0 0) 0x7f4400144940 con 0x7f440013f650 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43e4004eb0 con 0x7f440013f650 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f43fc001cb0 con 0x7f4400071c00 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): dump: 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:01.520 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755905974 0 0) 0x7f4400142cd0 con 0x7f44000a9d70 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4400144940 con 0x7f44000a9d70 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f43f0001490 con 0x7f440013f650 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.533+0000 7f43f5ffb640 10 monclient(hunting): dump: 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:01.521 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3676605349 0 0) 0x7f44001427b0 con 0x7f4400071c00 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 >> v1:172.21.15.195:6789/0 conn(0x7f44000a9d70 legacy=0x7f4400141fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 >> v1:172.21.15.157:6789/0 conn(0x7f440013f650 legacy=0x7f44000714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 1 monclient: found mon.c 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400144ad0 con 0x7f4400071c00 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:01.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f440776b640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f440776b640 10 monclient: _renew_subs 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f440776b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44001d3590 con 0x7f4400071c00 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f440776b640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f440776b640 10 monclient: _renew_subs 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f440776b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44001419e0 con 0x7f4400071c00 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f440776b640 1 librados: init done 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43fc001e10 con 0x7f4400071c00 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.534+0000 7f43f77fe640 4 set_mon_vals no callback set 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.535+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f43fc004e10 con 0x7f4400071c00 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.535+0000 7f43f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:01.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.535+0000 7f43f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.535+0000 7f43f5ffb640 10 monclient: dump: 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.536+0000 7f440776b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.536+0000 7f440776b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.536+0000 7f440776b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:01.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.536+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4400142e60 con 0x7f4400071c00 2024-03-29T11:16:01.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.536+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f43fc01ade0 con 0x7f4400071c00 2024-03-29T11:16:01.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.537+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f43fc084e70 con 0x7f4400071c00 2024-03-29T11:16:01.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.537+0000 7f43f5ffb640 10 client.4409.objecter ms_dispatch 0x7f4400000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:01.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.537+0000 7f43f5ffb640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:01.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.537+0000 7f43f5ffb640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:01.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.537+0000 7f4405507640 10 monclient: get_auth_request con 0x7f43e406dfb0 auth_method 0 2024-03-29T11:16:01.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.537+0000 7f43f5ffb640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:16:01.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.537+0000 7f43f5ffb640 10 client.4409.objecter ms_handle_connect 0x7f43e406dfb0 2024-03-29T11:16:01.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.547+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43fc050750 con 0x7f4400071c00 2024-03-29T11:16:01.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.547+0000 7f43f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:01.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.547+0000 7f43f5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:01.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.919+0000 7f440776b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:01.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.919+0000 7f440776b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:01.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.919+0000 7f440776b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:01.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.919+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4400075f10 con 0x7f4400071c00 2024-03-29T11:16:01.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.921+0000 7f43f5ffb640 1 -- 172.21.15.157:0/3149403650 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f43fc052680 con 0x7f4400071c00 2024-03-29T11:16:01.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.921+0000 7f43f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:01.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.921+0000 7f43f5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:01.899 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:16:01.899 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:01.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.925+0000 7f440776b640 10 librados: watch_flush enter 2024-03-29T11:16:01.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.926+0000 7f440776b640 10 librados: watch_flush exit 2024-03-29T11:16:01.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.926+0000 7f440776b640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:16:01.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.926+0000 7f440776b640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:16:01.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.926+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 >> v1:172.21.15.157:6816/1747058247 conn(0x7f43e406dfb0 legacy=0x7f43e4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:01.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.926+0000 7f440776b640 10 monclient: shutdown 2024-03-29T11:16:01.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.926+0000 7f440776b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:01.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.926+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 >> v1:172.21.15.157:6790/0 conn(0x7f4400071c00 legacy=0x7f44001d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:01.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.926+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 shutdown_connections 2024-03-29T11:16:01.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.926+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 >> 172.21.15.157:0/3149403650 conn(0x7f4400066a00 msgr2=0x7f440013e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:01.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.927+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 shutdown_connections 2024-03-29T11:16:01.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.927+0000 7f440776b640 1 -- 172.21.15.157:0/3149403650 wait complete. 2024-03-29T11:16:01.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:01.927+0000 7f440776b640 1 librados: shutdown 2024-03-29T11:16:01.920 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:08.728 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:08.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.901+0000 7fb724e25640 10 monclient: get_monmap_and_config 2024-03-29T11:16:08.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.901+0000 7fb724e25640 10 monclient: build_initial_monmap 2024-03-29T11:16:08.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.901+0000 7fb724e25640 10 monclient: monmap: 2024-03-29T11:16:08.879 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:08.879 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:08.879 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:16:08.902180+0000 2024-03-29T11:16:08.879 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:16:08.902180+0000 2024-03-29T11:16:08.879 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:08.879 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:08.879 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:08.880 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:16:08.880 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:16:08.880 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:08.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.901+0000 7fb724e25640 1 Processor -- start 2024-03-29T11:16:08.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.902+0000 7fb724e25640 1 -- start start 2024-03-29T11:16:08.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.902+0000 7fb724e25640 10 monclient: init 2024-03-29T11:16:08.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb724e25640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:08.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb724e25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:08.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb724e25640 10 monclient(hunting): picked mon.noname-c con 0x7fb7201422b0 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:08.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb724e25640 10 monclient(hunting): picked mon.noname-a con 0x7fb720142690 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:08.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb724e25640 10 monclient(hunting): picked mon.noname-b con 0x7fb7200b17d0 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:08.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb724e25640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb720142c80 con 0x7fb720142690 2024-03-29T11:16:08.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb724e25640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7200aae40 con 0x7fb7201422b0 2024-03-29T11:16:08.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb724e25640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72013fbd0 con 0x7fb7200b17d0 2024-03-29T11:16:08.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb724e25640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:08.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb724e25640 10 monclient(hunting): authenticate will time out at 1181.100080s 2024-03-29T11:16:08.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb71ed9d640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fb7200b17d0 0x7fb7200b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:60654/0 (socket says 172.21.15.157:60654) 2024-03-29T11:16:08.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb71ed9d640 1 -- 172.21.15.157:0/3996227925 learned_addr learned my addr 172.21.15.157:0/3996227925 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:08.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb71d59a640 1 -- 172.21.15.157:0/3996227925 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660145207 0 0) 0x7fb72013fbd0 con 0x7fb7200b17d0 2024-03-29T11:16:08.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb71d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:16:08.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb71d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:08.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb71d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:08.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb71d59a640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:16:08.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.903+0000 7fb71d59a640 1 -- 172.21.15.157:0/3996227925 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb708003520 con 0x7fb7200b17d0 2024-03-29T11:16:08.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 1 -- 172.21.15.157:0/3996227925 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850300083 0 0) 0x7fb708003520 con 0x7fb7200b17d0 2024-03-29T11:16:08.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 1 -- 172.21.15.157:0/3996227925 >> v1:172.21.15.195:6789/0 conn(0x7fb7201422b0 legacy=0x7fb72013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 1 -- 172.21.15.157:0/3996227925 >> v1:172.21.15.157:6789/0 conn(0x7fb720142690 legacy=0x7fb7200b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:16:08.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 1 monclient: found mon.noname-b 2024-03-29T11:16:08.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:08.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:16:08.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 1 -- 172.21.15.157:0/3996227925 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720151940 con 0x7fb7200b17d0 2024-03-29T11:16:08.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: _finish_auth 0 2024-03-29T11:16:08.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 1 -- 172.21.15.157:0/3996227925 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb718001a10 con 0x7fb7200b17d0 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: dump: 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:08.885 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb724e25640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb724e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 1 -- 172.21.15.157:0/3996227925 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb718001b70 con 0x7fb7200b17d0 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 1 -- 172.21.15.157:0/3996227925 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb718001f40 con 0x7fb7200b17d0 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb71d59a640 10 monclient: dump: 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:08.886 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb724e25640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.904+0000 7fb724e25640 4 set_mon_vals no callback set 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.905+0000 7fb724e25640 10 monclient: shutdown 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.905+0000 7fb724e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.905+0000 7fb724e25640 1 -- 172.21.15.157:0/3996227925 >> v1:172.21.15.157:6790/0 conn(0x7fb7200b17d0 legacy=0x7fb7200b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.905+0000 7fb724e25640 1 -- 172.21.15.157:0/3996227925 shutdown_connections 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.905+0000 7fb724e25640 1 -- 172.21.15.157:0/3996227925 >> 172.21.15.157:0/3996227925 conn(0x7fb720066a00 msgr2=0x7fb720066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.905+0000 7fb724e25640 1 -- 172.21.15.157:0/3996227925 shutdown_connections 2024-03-29T11:16:08.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.905+0000 7fb724e25640 1 -- 172.21.15.157:0/3996227925 wait complete. 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.906+0000 7fb724e25640 10 monclient: build_initial_monmap 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.906+0000 7fb724e25640 10 monclient: monmap: 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:16:08.888 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:08.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.907+0000 7fb724e25640 1 librados: starting msgr at 2024-03-29T11:16:08.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.907+0000 7fb724e25640 1 librados: starting objecter 2024-03-29T11:16:08.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.907+0000 7fb724e25640 1 Processor -- start 2024-03-29T11:16:08.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.907+0000 7fb724e25640 1 -- start start 2024-03-29T11:16:08.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.907+0000 7fb724e25640 1 librados: setting wanted keys 2024-03-29T11:16:08.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.907+0000 7fb724e25640 1 librados: calling monclient init 2024-03-29T11:16:08.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.907+0000 7fb724e25640 10 monclient: init 2024-03-29T11:16:08.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.908+0000 7fb724e25640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:08.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.908+0000 7fb724e25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:08.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.908+0000 7fb724e25640 10 monclient(hunting): picked mon.noname-b con 0x7fb7200b17d0 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.908+0000 7fb724e25640 10 monclient(hunting): picked mon.noname-c con 0x7fb7201422b0 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.908+0000 7fb724e25640 10 monclient(hunting): picked mon.noname-a con 0x7fb720142690 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.908+0000 7fb724e25640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb720070e30 con 0x7fb720142690 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.908+0000 7fb724e25640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7200b0e60 con 0x7fb7200b17d0 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.908+0000 7fb724e25640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb720071080 con 0x7fb7201422b0 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.908+0000 7fb724e25640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.908+0000 7fb724e25640 10 monclient(hunting): authenticate will time out at 1181.105512s 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb71dd9b640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fb7201422b0 0x7fb72006ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:33042/0 (socket says 172.21.15.157:33042) 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb71dd9b640 1 -- 172.21.15.157:0/1743741249 learned_addr learned my addr 172.21.15.157:0/1743741249 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 client.?.objecter ms_handle_connect 0x7fb7201422b0 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 client.?.objecter ms_handle_connect 0x7fb720142690 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:08.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 client.?.objecter ms_handle_connect 0x7fb7200b17d0 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154634843 0 0) 0x7fb720071080 con 0x7fb7201422b0 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb700004eb0 con 0x7fb7201422b0 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539171599 0 0) 0x7fb720070e30 con 0x7fb720142690 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.909+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb720071080 con 0x7fb720142690 2024-03-29T11:16:08.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305489631 0 0) 0x7fb7200b0e60 con 0x7fb7200b17d0 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb720070e30 con 0x7fb7200b17d0 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1197357725 0 0) 0x7fb720071080 con 0x7fb720142690 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7200b0e60 con 0x7fb720142690 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 752167866 0 0) 0x7fb700004eb0 con 0x7fb7201422b0 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb720071080 con 0x7fb7201422b0 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb718001e80 con 0x7fb720142690 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): dump: 2024-03-29T11:16:08.897 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb714001910 con 0x7fb7201422b0 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:08.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 10 monclient(hunting): dump: 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1589646317 0 0) 0x7fb7200b0e60 con 0x7fb720142690 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.910+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 >> v1:172.21.15.157:6790/0 conn(0x7fb7201422b0 legacy=0x7fb72006ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 >> v1:172.21.15.195:6789/0 conn(0x7fb7200b17d0 legacy=0x7fb72013be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb70effd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb70effd640 1 monclient: found mon.a 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb70effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb70effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72006df20 con 0x7fb720142690 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb70effd640 10 monclient: _finish_auth 0 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb70effd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb70effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb724e25640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb724e25640 10 monclient: _renew_subs 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb724e25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb724e25640 1 -- 172.21.15.157:0/1743741249 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72006e110 con 0x7fb720142690 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb724e25640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:08.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb724e25640 10 monclient: _renew_subs 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb724e25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb724e25640 1 -- 172.21.15.157:0/1743741249 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72013c770 con 0x7fb720142690 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.911+0000 7fb724e25640 1 librados: init done 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.912+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb718001fe0 con 0x7fb720142690 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.912+0000 7fb70effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.912+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb7180015c0 con 0x7fb720142690 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.912+0000 7fb70effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.912+0000 7fb70effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.912+0000 7fb70effd640 10 monclient: dump: 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:08.901 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.912+0000 7fb71cd99640 4 set_mon_vals no callback set 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.912+0000 7fb724e25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.913+0000 7fb724e25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.913+0000 7fb724e25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.913+0000 7fb724e25640 1 -- 172.21.15.157:0/1743741249 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb720075f10 con 0x7fb720142690 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.915+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fb71801ac80 con 0x7fb720142690 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.916+0000 7fb71e59c640 10 monclient: get_auth_request con 0x7fb70006df90 auth_method 0 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.916+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fb718084d40 con 0x7fb720142690 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.916+0000 7fb70effd640 10 client.4404.objecter ms_dispatch 0x7fb720000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:08.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.916+0000 7fb70effd640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:08.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.916+0000 7fb70effd640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:08.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.916+0000 7fb70effd640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-29T11:16:08.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.916+0000 7fb70effd640 10 client.4404.objecter ms_handle_connect 0x7fb70006df90 2024-03-29T11:16:08.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.924+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb718050450 con 0x7fb720142690 2024-03-29T11:16:08.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.924+0000 7fb70effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:08.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:08.924+0000 7fb70effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:09.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.254+0000 7fb724e25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:09.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.254+0000 7fb724e25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:09.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.254+0000 7fb724e25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:16:09.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.254+0000 7fb724e25640 1 -- 172.21.15.157:0/1743741249 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7200b2270 con 0x7fb720142690 2024-03-29T11:16:09.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.255+0000 7fb70effd640 1 -- 172.21.15.157:0/1743741249 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fb718052550 con 0x7fb720142690 2024-03-29T11:16:09.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.255+0000 7fb70effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:09.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.255+0000 7fb70effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:09.233 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:16:09.233 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:09.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.258+0000 7fb724e25640 10 librados: watch_flush enter 2024-03-29T11:16:09.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.258+0000 7fb724e25640 10 librados: watch_flush exit 2024-03-29T11:16:09.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.258+0000 7fb724e25640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-29T11:16:09.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.258+0000 7fb724e25640 10 client.4404.objecter successfully canceled tick 2024-03-29T11:16:09.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.259+0000 7fb724e25640 1 -- 172.21.15.157:0/1743741249 >> v1:172.21.15.157:6816/1747058247 conn(0x7fb70006df90 legacy=0x7fb700070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:09.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.259+0000 7fb724e25640 10 monclient: shutdown 2024-03-29T11:16:09.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.259+0000 7fb724e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:09.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.259+0000 7fb724e25640 1 -- 172.21.15.157:0/1743741249 >> v1:172.21.15.157:6789/0 conn(0x7fb720142690 legacy=0x7fb72006f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:09.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.260+0000 7fb724e25640 1 -- 172.21.15.157:0/1743741249 shutdown_connections 2024-03-29T11:16:09.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.260+0000 7fb724e25640 1 -- 172.21.15.157:0/1743741249 >> 172.21.15.157:0/1743741249 conn(0x7fb720066a00 msgr2=0x7fb72013d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:09.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.261+0000 7fb724e25640 1 -- 172.21.15.157:0/1743741249 shutdown_connections 2024-03-29T11:16:09.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.261+0000 7fb724e25640 1 -- 172.21.15.157:0/1743741249 wait complete. 2024-03-29T11:16:09.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:09.262+0000 7fb724e25640 1 librados: shutdown 2024-03-29T11:16:09.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:16.065 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:16.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.233+0000 7f356b020640 10 monclient: get_monmap_and_config 2024-03-29T11:16:16.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.233+0000 7f356b020640 10 monclient: build_initial_monmap 2024-03-29T11:16:16.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.234+0000 7f356b020640 10 monclient: monmap: 2024-03-29T11:16:16.211 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:16.211 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:16.211 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:16:16.235133+0000 2024-03-29T11:16:16.212 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:16:16.235133+0000 2024-03-29T11:16:16.212 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:16.212 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:16.212 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:16.212 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:16:16.212 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:16:16.212 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:16.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.235+0000 7f356b020640 1 Processor -- start 2024-03-29T11:16:16.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.237+0000 7f356b020640 1 -- start start 2024-03-29T11:16:16.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.237+0000 7f356b020640 10 monclient: init 2024-03-29T11:16:16.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.239+0000 7f356b020640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:16.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.239+0000 7f356b020640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:16.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.239+0000 7f356b020640 10 monclient(hunting): picked mon.noname-c con 0x7f355c037230 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:16.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.239+0000 7f356b020640 10 monclient(hunting): picked mon.noname-a con 0x7f355c0c9140 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:16.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.239+0000 7f356b020640 10 monclient(hunting): picked mon.noname-b con 0x7f355c0c95a0 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:16.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.239+0000 7f356b020640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c02c150 con 0x7f355c0c9140 2024-03-29T11:16:16.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.239+0000 7f356b020640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c02c430 con 0x7f355c037230 2024-03-29T11:16:16.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.239+0000 7f356b020640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c02afb0 con 0x7f355c0c95a0 2024-03-29T11:16:16.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.239+0000 7f356b020640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:16.217 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.239+0000 7f356b020640 10 monclient(hunting): authenticate will time out at 1188.436039s 2024-03-29T11:16:16.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.242+0000 7f356a01e640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f355c037230 0x7f355c02a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:43907/0 (socket says 172.21.15.157:43907) 2024-03-29T11:16:16.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.242+0000 7f356a01e640 1 -- 172.21.15.157:0/1347426191 learned_addr learned my addr 172.21.15.157:0/1347426191 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:16.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862122506 0 0) 0x7f355c02afb0 con 0x7f355c0c95a0 2024-03-29T11:16:16.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 10 monclient(hunting): _negotiate 2024-03-29T11:16:16.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:16.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:16.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:16:16.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3550003520 con 0x7f355c0c95a0 2024-03-29T11:16:16.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997155082 0 0) 0x7f355c02c150 con 0x7f355c0c9140 2024-03-29T11:16:16.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 10 monclient(hunting): _negotiate 2024-03-29T11:16:16.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:16.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:16.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:16:16.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.243+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f355c02afb0 con 0x7f355c0c9140 2024-03-29T11:16:16.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1474997125 0 0) 0x7f355c02afb0 con 0x7f355c0c9140 2024-03-29T11:16:16.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 >> v1:172.21.15.157:6790/0 conn(0x7f355c0c95a0 legacy=0x7f355c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 >> v1:172.21.15.195:6789/0 conn(0x7f355c037230 legacy=0x7f355c02a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 10 monclient: _finish_hunting 0 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 1 monclient: found mon.noname-a 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0db5c0 con 0x7f355c0c9140 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 10 monclient: _finish_auth 0 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 10 monclient: _check_auth_tickets 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f356404f730 con 0x7f355c0c9140 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:16.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356901c640 10 monclient: dump: 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356b020640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:16:16.226 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.244+0000 7f356b020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.247+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35640008d0 con 0x7f355c0c9140 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.247+0000 7f356901c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.247+0000 7f356901c640 1 -- 172.21.15.157:0/1347426191 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f35640758b0 con 0x7f355c0c9140 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.247+0000 7f356901c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.247+0000 7f356901c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.247+0000 7f356901c640 10 monclient: dump: 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:16.228 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.247+0000 7f356b020640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.247+0000 7f356b020640 4 set_mon_vals no callback set 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.248+0000 7f356b020640 10 monclient: shutdown 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.248+0000 7f356b020640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.248+0000 7f356b020640 1 -- 172.21.15.157:0/1347426191 >> v1:172.21.15.157:6789/0 conn(0x7f355c0c9140 legacy=0x7f355c0cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.248+0000 7f356b020640 1 -- 172.21.15.157:0/1347426191 shutdown_connections 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.248+0000 7f356b020640 1 -- 172.21.15.157:0/1347426191 >> 172.21.15.157:0/1347426191 conn(0x7f355c03a5e0 msgr2=0x7f355c036830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.248+0000 7f356b020640 1 -- 172.21.15.157:0/1347426191 shutdown_connections 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.248+0000 7f356b020640 1 -- 172.21.15.157:0/1347426191 wait complete. 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.249+0000 7f356b020640 10 monclient: build_initial_monmap 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.249+0000 7f356b020640 10 monclient: monmap: 2024-03-29T11:16:16.229 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.250+0000 7f356b020640 1 librados: starting msgr at 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.250+0000 7f356b020640 1 librados: starting objecter 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.250+0000 7f356b020640 1 Processor -- start 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.250+0000 7f356b020640 1 -- start start 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.250+0000 7f356b020640 1 librados: setting wanted keys 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.250+0000 7f356b020640 1 librados: calling monclient init 2024-03-29T11:16:16.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.250+0000 7f356b020640 10 monclient: init 2024-03-29T11:16:16.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356b020640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356b020640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356b020640 10 monclient(hunting): picked mon.noname-c con 0x7f355c037230 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356b020640 10 monclient(hunting): picked mon.noname-b con 0x7f355c0c9140 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356b020640 10 monclient(hunting): picked mon.noname-a con 0x7f355c0c95a0 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356b020640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0ba830 con 0x7f355c0c95a0 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356b020640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0cb540 con 0x7f355c0c9140 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356b020640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0b80f0 con 0x7f355c037230 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356b020640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356b020640 10 monclient(hunting): authenticate will time out at 1188.448392s 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356a01e640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f355c037230 0x7f355c0b9130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:40274/0 (socket says 172.21.15.157:40274) 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.251+0000 7f356a01e640 1 -- 172.21.15.157:0/1301604593 learned_addr learned my addr 172.21.15.157:0/1301604593 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.254+0000 7f355affd640 10 client.?.objecter ms_handle_connect 0x7f355c037230 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.254+0000 7f355affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:16.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.254+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781929090 0 0) 0x7f355c0b80f0 con 0x7f355c037230 2024-03-29T11:16:16.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.254+0000 7f355affd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:16.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.254+0000 7f355affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:16.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.254+0000 7f355affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:16.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.254+0000 7f355affd640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:16:16.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.254+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3548004eb0 con 0x7f355c037230 2024-03-29T11:16:16.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3709597558 0 0) 0x7f3548004eb0 con 0x7f355c037230 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f355c0b80f0 con 0x7f355c037230 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3560001930 con 0x7f355c037230 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 monclient(hunting): dump: 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:16.236 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 client.?.objecter ms_handle_connect 0x7f355c0c9140 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 client.?.objecter ms_handle_connect 0x7f355c0c95a0 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 869126497 0 0) 0x7f355c0b80f0 con 0x7f355c037230 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 >> v1:172.21.15.195:6789/0 conn(0x7f355c0c9140 legacy=0x7f355c0b72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 >> v1:172.21.15.157:6789/0 conn(0x7f355c0c95a0 legacy=0x7f355c0b79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 1 monclient: found mon.c 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:16.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0cc0d0 con 0x7f355c037230 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 monclient: _finish_auth 0 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f355affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f356b020640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f356b020640 10 monclient: _renew_subs 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f356b020640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.255+0000 7f356b020640 1 -- 172.21.15.157:0/1301604593 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c0cc3b0 con 0x7f355c037230 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f356b020640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f356b020640 10 monclient: _renew_subs 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f356b020640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f356b020640 1 -- 172.21.15.157:0/1301604593 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c033880 con 0x7f355c037230 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f356b020640 1 librados: init done 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35600047e0 con 0x7f355c037230 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f355affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3560004c30 con 0x7f355c037230 2024-03-29T11:16:16.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f355affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f356881b640 4 set_mon_vals no callback set 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f355affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.256+0000 7f355affd640 10 monclient: dump: 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.258+0000 7f356b020640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.258+0000 7f356b020640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:16.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.258+0000 7f356b020640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:16.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.258+0000 7f356b020640 1 -- 172.21.15.157:0/1301604593 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354c005080 con 0x7f355c037230 2024-03-29T11:16:16.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.258+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f3560004d90 con 0x7f355c037230 2024-03-29T11:16:16.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.259+0000 7f356981d640 10 monclient: get_auth_request con 0x7f354806dbb0 auth_method 0 2024-03-29T11:16:16.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.259+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f3560084ba0 con 0x7f355c037230 2024-03-29T11:16:16.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.259+0000 7f355affd640 10 client.4421.objecter ms_dispatch 0x7f3564000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:16.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.259+0000 7f355affd640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:16.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.259+0000 7f355affd640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:16.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.259+0000 7f355affd640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T11:16:16.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.259+0000 7f355affd640 10 client.4421.objecter ms_handle_connect 0x7f354806dbb0 2024-03-29T11:16:16.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.266+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3560050480 con 0x7f355c037230 2024-03-29T11:16:16.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.266+0000 7f355affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:16.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.266+0000 7f355affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:16.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.586+0000 7f356b020640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:16.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.587+0000 7f356b020640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:16.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.587+0000 7f356b020640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:16.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.587+0000 7f356b020640 1 -- 172.21.15.157:0/1301604593 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f354c005330 con 0x7f355c037230 2024-03-29T11:16:16.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.587+0000 7f355affd640 1 -- 172.21.15.157:0/1301604593 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f35600523b0 con 0x7f355c037230 2024-03-29T11:16:16.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.587+0000 7f355affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:16.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.587+0000 7f355affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:16.567 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:16:16.567 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:16.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.593+0000 7f356b020640 10 librados: watch_flush enter 2024-03-29T11:16:16.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.593+0000 7f356b020640 10 librados: watch_flush exit 2024-03-29T11:16:16.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.593+0000 7f356b020640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T11:16:16.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.593+0000 7f356b020640 10 client.4421.objecter successfully canceled tick 2024-03-29T11:16:16.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.593+0000 7f356b020640 1 -- 172.21.15.157:0/1301604593 >> v1:172.21.15.157:6816/1747058247 conn(0x7f354806dbb0 legacy=0x7f3548070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:16.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.593+0000 7f356b020640 10 monclient: shutdown 2024-03-29T11:16:16.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.593+0000 7f356b020640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:16.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.593+0000 7f356b020640 1 -- 172.21.15.157:0/1301604593 >> v1:172.21.15.157:6790/0 conn(0x7f355c037230 legacy=0x7f355c0b9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:16.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.594+0000 7f356b020640 1 -- 172.21.15.157:0/1301604593 shutdown_connections 2024-03-29T11:16:16.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.594+0000 7f356b020640 1 -- 172.21.15.157:0/1301604593 >> 172.21.15.157:0/1301604593 conn(0x7f355c03a5e0 msgr2=0x7f355c029ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:16.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.594+0000 7f356b020640 1 -- 172.21.15.157:0/1301604593 shutdown_connections 2024-03-29T11:16:16.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.594+0000 7f356b020640 1 -- 172.21.15.157:0/1301604593 wait complete. 2024-03-29T11:16:16.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:16.594+0000 7f356b020640 1 librados: shutdown 2024-03-29T11:16:16.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:23.395 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:23.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.574+0000 7f37f4217640 10 monclient: get_monmap_and_config 2024-03-29T11:16:23.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.574+0000 7f37f4217640 10 monclient: build_initial_monmap 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.574+0000 7f37f4217640 10 monclient: monmap: 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:16:23.576052+0000 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:16:23.576052+0000 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:16:23.553 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:23.557 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.580+0000 7f37f4217640 1 Processor -- start 2024-03-29T11:16:23.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.581+0000 7f37f4217640 1 -- start start 2024-03-29T11:16:23.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.581+0000 7f37f4217640 10 monclient: init 2024-03-29T11:16:23.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.582+0000 7f37f4217640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:23.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.582+0000 7f37f4217640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:23.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.582+0000 7f37f4217640 10 monclient(hunting): picked mon.noname-c con 0x7f37ec0a9750 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:23.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.585+0000 7f37f4217640 10 monclient(hunting): picked mon.noname-a con 0x7f37ec13e1e0 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:23.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.585+0000 7f37f17b2640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f37ec13e1e0 0x7f37ec0ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:53221/0 (socket says 172.21.15.157:53221) 2024-03-29T11:16:23.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.585+0000 7f37f17b2640 1 -- 172.21.15.157:0/3027656162 learned_addr learned my addr 172.21.15.157:0/3027656162 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:23.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.585+0000 7f37f4217640 10 monclient(hunting): picked mon.noname-b con 0x7f37ec071c30 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:23.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.585+0000 7f37f4217640 1 -- 172.21.15.157:0/3027656162 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec0ab300 con 0x7f37ec13e1e0 2024-03-29T11:16:23.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.586+0000 7f37f4217640 1 -- 172.21.15.157:0/3027656162 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec0ac2f0 con 0x7f37ec0a9750 2024-03-29T11:16:23.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.586+0000 7f37f4217640 1 -- 172.21.15.157:0/3027656162 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec0ad8d0 con 0x7f37ec071c30 2024-03-29T11:16:23.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.586+0000 7f37f4217640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:23.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.586+0000 7f37f4217640 10 monclient(hunting): authenticate will time out at 1195.783147s 2024-03-29T11:16:23.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.588+0000 7f37f0fb1640 1 -- 172.21.15.157:0/3027656162 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616677644 0 0) 0x7f37ec0ab300 con 0x7f37ec13e1e0 2024-03-29T11:16:23.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.588+0000 7f37f0fb1640 10 monclient(hunting): _negotiate 2024-03-29T11:16:23.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.588+0000 7f37f0fb1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:23.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.588+0000 7f37f0fb1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:23.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.588+0000 7f37f0fb1640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:16:23.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.588+0000 7f37f0fb1640 1 -- 172.21.15.157:0/3027656162 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d8003520 con 0x7f37ec13e1e0 2024-03-29T11:16:23.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.588+0000 7f37f0fb1640 1 -- 172.21.15.157:0/3027656162 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 685309545 0 0) 0x7f37d8003520 con 0x7f37ec13e1e0 2024-03-29T11:16:23.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.588+0000 7f37f0fb1640 1 -- 172.21.15.157:0/3027656162 >> v1:172.21.15.157:6790/0 conn(0x7f37ec071c30 legacy=0x7f37ec0ace20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:23.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 1 -- 172.21.15.157:0/3027656162 >> v1:172.21.15.195:6789/0 conn(0x7f37ec0a9750 legacy=0x7f37ec13dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 10 monclient: _finish_hunting 0 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 1 monclient: found mon.noname-a 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 1 -- 172.21.15.157:0/3027656162 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec0ada60 con 0x7f37ec13e1e0 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 10 monclient: _finish_auth 0 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 10 monclient: _check_auth_tickets 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f4217640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f4217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 1 -- 172.21.15.157:0/3027656162 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f37dc001690 con 0x7f37ec13e1e0 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:23.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f0fb1640 10 monclient: dump: 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:23.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.590+0000 7f37f4217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:23.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.592+0000 7f37f0fb1640 1 -- 172.21.15.157:0/3027656162 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37dc001860 con 0x7f37ec13e1e0 2024-03-29T11:16:23.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.592+0000 7f37f0fb1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:23.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.592+0000 7f37f4217640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:23.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.592+0000 7f37f4217640 4 set_mon_vals no callback set 2024-03-29T11:16:23.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.593+0000 7f37f4217640 10 monclient: shutdown 2024-03-29T11:16:23.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.593+0000 7f37f4217640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:23.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.593+0000 7f37f4217640 1 -- 172.21.15.157:0/3027656162 >> v1:172.21.15.157:6789/0 conn(0x7f37ec13e1e0 legacy=0x7f37ec0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:23.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.593+0000 7f37f4217640 1 -- 172.21.15.157:0/3027656162 shutdown_connections 2024-03-29T11:16:23.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.593+0000 7f37f4217640 1 -- 172.21.15.157:0/3027656162 >> 172.21.15.157:0/3027656162 conn(0x7f37ec06b660 msgr2=0x7f37ec06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:23.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.593+0000 7f37f4217640 1 -- 172.21.15.157:0/3027656162 shutdown_connections 2024-03-29T11:16:23.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.594+0000 7f37f4217640 1 -- 172.21.15.157:0/3027656162 wait complete. 2024-03-29T11:16:23.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.595+0000 7f37f4217640 10 monclient: build_initial_monmap 2024-03-29T11:16:23.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.595+0000 7f37f4217640 10 monclient: monmap: 2024-03-29T11:16:23.575 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:23.575 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:23.575 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:16:23.575 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:16:23.575 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:23.575 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.595+0000 7f37f4217640 1 librados: starting msgr at 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.595+0000 7f37f4217640 1 librados: starting objecter 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.595+0000 7f37f4217640 1 Processor -- start 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.597+0000 7f37f4217640 1 -- start start 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.597+0000 7f37f4217640 1 librados: setting wanted keys 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.597+0000 7f37f4217640 1 librados: calling monclient init 2024-03-29T11:16:23.576 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.597+0000 7f37f4217640 10 monclient: init 2024-03-29T11:16:23.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.598+0000 7f37f4217640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:23.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.598+0000 7f37f4217640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:16:23.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.598+0000 7f37f4217640 10 monclient(hunting): picked mon.noname-c con 0x7f37ec071c30 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:23.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.598+0000 7f37f4217640 10 monclient(hunting): picked mon.noname-b con 0x7f37ec0a9750 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:23.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.598+0000 7f37f4217640 10 monclient(hunting): picked mon.noname-a con 0x7f37ec13e1e0 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:23.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.598+0000 7f37f4217640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec141e80 con 0x7f37ec13e1e0 2024-03-29T11:16:23.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.599+0000 7f37f4217640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec1355e0 con 0x7f37ec0a9750 2024-03-29T11:16:23.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.599+0000 7f37f4217640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec136410 con 0x7f37ec071c30 2024-03-29T11:16:23.577 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.599+0000 7f37f4217640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:23.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.599+0000 7f37f4217640 10 monclient(hunting): authenticate will time out at 1195.795979s 2024-03-29T11:16:23.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.604+0000 7f37f17b2640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f37ec0a9750 0x7f37ec1d2d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:44033/0 (socket says 172.21.15.157:44033) 2024-03-29T11:16:23.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.604+0000 7f37f17b2640 1 -- 172.21.15.157:0/3686206900 learned_addr learned my addr 172.21.15.157:0/3686206900 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:23.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.604+0000 7f37e27fc640 10 client.?.objecter ms_handle_connect 0x7f37ec0a9750 2024-03-29T11:16:23.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.604+0000 7f37e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:23.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.604+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983642546 0 0) 0x7f37ec1355e0 con 0x7f37ec0a9750 2024-03-29T11:16:23.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.604+0000 7f37e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:23.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.604+0000 7f37e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:23.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.604+0000 7f37e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:23.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.604+0000 7f37e27fc640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:16:23.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.604+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d0004eb0 con 0x7f37ec0a9750 2024-03-29T11:16:23.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 client.?.objecter ms_handle_connect 0x7f37ec071c30 2024-03-29T11:16:23.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:23.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 client.?.objecter ms_handle_connect 0x7f37ec13e1e0 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1770709693 0 0) 0x7f37d0004eb0 con 0x7f37ec0a9750 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37ec1355e0 con 0x7f37ec0a9750 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265972076 0 0) 0x7f37ec136410 con 0x7f37ec071c30 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d0004eb0 con 0x7f37ec071c30 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f37dc0049c0 con 0x7f37ec0a9750 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 monclient(hunting): dump: 2024-03-29T11:16:23.585 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.605+0000 7f37e27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1852116357 0 0) 0x7f37ec1355e0 con 0x7f37ec0a9750 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 >> v1:172.21.15.157:6790/0 conn(0x7f37ec071c30 legacy=0x7f37ec135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 >> v1:172.21.15.157:6789/0 conn(0x7f37ec13e1e0 legacy=0x7f37ec1dd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:23.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 1 monclient: found mon.b 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec142920 con 0x7f37ec0a9750 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37f4217640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37f4217640 10 monclient: _renew_subs 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37f4217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37f4217640 1 -- 172.21.15.157:0/3686206900 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec142310 con 0x7f37ec0a9750 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37f4217640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37f4217640 10 monclient: _renew_subs 2024-03-29T11:16:23.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37f4217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:23.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37f4217640 1 -- 172.21.15.157:0/3686206900 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ec1d3620 con 0x7f37ec0a9750 2024-03-29T11:16:23.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.606+0000 7f37f4217640 1 librados: init done 2024-03-29T11:16:23.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.607+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37dc001270 con 0x7f37ec0a9750 2024-03-29T11:16:23.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.607+0000 7f37e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:23.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.607+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f37dc004e90 con 0x7f37ec0a9750 2024-03-29T11:16:23.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.608+0000 7f37e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:23.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.608+0000 7f37e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.608+0000 7f37e27fc640 10 monclient: dump: 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.608+0000 7f37e3fff640 4 set_mon_vals no callback set 2024-03-29T11:16:23.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.610+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f37dc004ff0 con 0x7f37ec0a9750 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.611+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f37dc084d70 con 0x7f37ec0a9750 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.611+0000 7f37f4217640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.611+0000 7f37f1fb3640 10 monclient: get_auth_request con 0x7f37d006ddf0 auth_method 0 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.612+0000 7f37e27fc640 10 client.4390.objecter ms_dispatch 0x7f37ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.612+0000 7f37e27fc640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.612+0000 7f37f4217640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.612+0000 7f37f4217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.612+0000 7f37e27fc640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.612+0000 7f37f4217640 1 -- 172.21.15.157:0/3686206900 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d4005080 con 0x7f37ec0a9750 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.612+0000 7f37e27fc640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T11:16:23.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.613+0000 7f37e27fc640 10 client.4390.objecter ms_handle_connect 0x7f37d006ddf0 2024-03-29T11:16:23.598 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.620+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37dc050650 con 0x7f37ec0a9750 2024-03-29T11:16:23.598 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.621+0000 7f37e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:23.598 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.621+0000 7f37e27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:23.916 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.938+0000 7f37f4217640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:23.916 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.938+0000 7f37f4217640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:23.916 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.938+0000 7f37f4217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:23.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.938+0000 7f37f4217640 1 -- 172.21.15.157:0/3686206900 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37d4005330 con 0x7f37ec0a9750 2024-03-29T11:16:23.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.942+0000 7f37e27fc640 1 -- 172.21.15.157:0/3686206900 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f37dc052580 con 0x7f37ec0a9750 2024-03-29T11:16:23.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.942+0000 7f37e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:23.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.942+0000 7f37e27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:23.922 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:16:23.922 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:23.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.948+0000 7f37f4217640 10 librados: watch_flush enter 2024-03-29T11:16:23.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.949+0000 7f37f4217640 10 librados: watch_flush exit 2024-03-29T11:16:23.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.949+0000 7f37f4217640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T11:16:23.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.949+0000 7f37f4217640 10 client.4390.objecter successfully canceled tick 2024-03-29T11:16:23.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.949+0000 7f37f4217640 1 -- 172.21.15.157:0/3686206900 >> v1:172.21.15.157:6816/1747058247 conn(0x7f37d006ddf0 legacy=0x7f37d00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:23.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.949+0000 7f37f4217640 10 monclient: shutdown 2024-03-29T11:16:23.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.949+0000 7f37f4217640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:23.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.949+0000 7f37f4217640 1 -- 172.21.15.157:0/3686206900 >> v1:172.21.15.195:6789/0 conn(0x7f37ec0a9750 legacy=0x7f37ec1d2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:23.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.950+0000 7f37f4217640 1 -- 172.21.15.157:0/3686206900 shutdown_connections 2024-03-29T11:16:23.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.951+0000 7f37f4217640 1 -- 172.21.15.157:0/3686206900 >> 172.21.15.157:0/3686206900 conn(0x7f37ec06b660 msgr2=0x7f37ec06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:23.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.952+0000 7f37f4217640 1 -- 172.21.15.157:0/3686206900 shutdown_connections 2024-03-29T11:16:23.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.954+0000 7f37f4217640 1 -- 172.21.15.157:0/3686206900 wait complete. 2024-03-29T11:16:23.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:23.955+0000 7f37f4217640 1 librados: shutdown 2024-03-29T11:16:23.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:30.756 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:30.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.926+0000 7f6c9bed0640 10 monclient: get_monmap_and_config 2024-03-29T11:16:30.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.926+0000 7f6c9bed0640 10 monclient: build_initial_monmap 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.926+0000 7f6c9bed0640 10 monclient: monmap: 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:16:30.927378+0000 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:16:30.927378+0000 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:16:30.904 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:30.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.927+0000 7f6c9bed0640 1 Processor -- start 2024-03-29T11:16:30.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.927+0000 7f6c9bed0640 1 -- start start 2024-03-29T11:16:30.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.927+0000 7f6c9bed0640 10 monclient: init 2024-03-29T11:16:30.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.928+0000 7f6c9bed0640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:30.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.928+0000 7f6c9bed0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:30.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.930+0000 7f6c9bed0640 10 monclient(hunting): picked mon.noname-c con 0x7f6c940b1f60 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:30.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.930+0000 7f6c9bed0640 10 monclient(hunting): picked mon.noname-a con 0x7f6c94071eb0 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:30.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.930+0000 7f6c9bed0640 10 monclient(hunting): picked mon.noname-b con 0x7f6c94143d10 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:30.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.930+0000 7f6c9bed0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9406ef80 con 0x7f6c94071eb0 2024-03-29T11:16:30.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.930+0000 7f6c9bed0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9406f260 con 0x7f6c940b1f60 2024-03-29T11:16:30.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.930+0000 7f6c9bed0640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c940b40c0 con 0x7f6c94143d10 2024-03-29T11:16:30.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.930+0000 7f6c9bed0640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:30.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.930+0000 7f6c9bed0640 10 monclient(hunting): authenticate will time out at 1203.127534s 2024-03-29T11:16:30.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.934+0000 7f6c9946b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6c94071eb0 0x7f6c94146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:42373/0 (socket says 172.21.15.157:42373) 2024-03-29T11:16:30.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.934+0000 7f6c9946b640 1 -- 172.21.15.157:0/2102827038 learned_addr learned my addr 172.21.15.157:0/2102827038 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:30.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.940+0000 7f6c98c6a640 1 -- 172.21.15.157:0/2102827038 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230679298 0 0) 0x7f6c9406ef80 con 0x7f6c94071eb0 2024-03-29T11:16:30.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.940+0000 7f6c98c6a640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.940+0000 7f6c98c6a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.940+0000 7f6c98c6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.940+0000 7f6c98c6a640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:16:30.918 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.940+0000 7f6c98c6a640 1 -- 172.21.15.157:0/2102827038 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c80003520 con 0x7f6c94071eb0 2024-03-29T11:16:30.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.940+0000 7f6c98c6a640 1 -- 172.21.15.157:0/2102827038 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1966909860 0 0) 0x7f6c80003520 con 0x7f6c94071eb0 2024-03-29T11:16:30.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.940+0000 7f6c98c6a640 1 -- 172.21.15.157:0/2102827038 >> v1:172.21.15.157:6790/0 conn(0x7f6c94143d10 legacy=0x7f6c94146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:30.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 1 -- 172.21.15.157:0/2102827038 >> v1:172.21.15.195:6789/0 conn(0x7f6c940b1f60 legacy=0x7f6c94143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:30.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: _finish_hunting 0 2024-03-29T11:16:30.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 1 monclient: found mon.noname-a 2024-03-29T11:16:30.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:30.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:16:30.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 1 -- 172.21.15.157:0/2102827038 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c94155d70 con 0x7f6c94071eb0 2024-03-29T11:16:30.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: _finish_auth 0 2024-03-29T11:16:30.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: _check_auth_tickets 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 1 -- 172.21.15.157:0/2102827038 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6c84001690 con 0x7f6c94071eb0 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: dump: 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c9bed0640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:16:30.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c9bed0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 1 -- 172.21.15.157:0/2102827038 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c84001860 con 0x7f6c94071eb0 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 1 -- 172.21.15.157:0/2102827038 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6c84001cb0 con 0x7f6c94071eb0 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c9bed0640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c9bed0640 4 set_mon_vals no callback set 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.941+0000 7f6c98c6a640 10 monclient: dump: 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:30.921 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:30.922 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:30.922 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.942+0000 7f6c9bed0640 10 monclient: shutdown 2024-03-29T11:16:30.922 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.942+0000 7f6c9bed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:30.922 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.942+0000 7f6c9bed0640 1 -- 172.21.15.157:0/2102827038 >> v1:172.21.15.157:6789/0 conn(0x7f6c94071eb0 legacy=0x7f6c94146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.942+0000 7f6c9bed0640 1 -- 172.21.15.157:0/2102827038 shutdown_connections 2024-03-29T11:16:30.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.942+0000 7f6c9bed0640 1 -- 172.21.15.157:0/2102827038 >> 172.21.15.157:0/2102827038 conn(0x7f6c94066a00 msgr2=0x7f6c94066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:30.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.945+0000 7f6c9bed0640 1 -- 172.21.15.157:0/2102827038 shutdown_connections 2024-03-29T11:16:30.923 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.946+0000 7f6c9bed0640 1 -- 172.21.15.157:0/2102827038 wait complete. 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.947+0000 7f6c9bed0640 10 monclient: build_initial_monmap 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.947+0000 7f6c9bed0640 10 monclient: monmap: 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:16:30.925 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:30.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.948+0000 7f6c9bed0640 1 librados: starting msgr at 2024-03-29T11:16:30.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.948+0000 7f6c9bed0640 1 librados: starting objecter 2024-03-29T11:16:30.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.948+0000 7f6c9bed0640 1 Processor -- start 2024-03-29T11:16:30.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.948+0000 7f6c9bed0640 1 -- start start 2024-03-29T11:16:30.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.948+0000 7f6c9bed0640 1 librados: setting wanted keys 2024-03-29T11:16:30.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.948+0000 7f6c9bed0640 1 librados: calling monclient init 2024-03-29T11:16:30.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.948+0000 7f6c9bed0640 10 monclient: init 2024-03-29T11:16:30.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.949+0000 7f6c9bed0640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:30.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.949+0000 7f6c9bed0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:30.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.949+0000 7f6c9bed0640 10 monclient(hunting): picked mon.noname-a con 0x7f6c940b1f60 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:30.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.950+0000 7f6c9bed0640 10 monclient(hunting): picked mon.noname-c con 0x7f6c94143d10 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:30.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.950+0000 7f6c9bed0640 10 monclient(hunting): picked mon.noname-b con 0x7f6c941419e0 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:30.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.950+0000 7f6c9bed0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c941548b0 con 0x7f6c940b1f60 2024-03-29T11:16:30.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.950+0000 7f6c9bed0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c94154b90 con 0x7f6c941419e0 2024-03-29T11:16:30.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.950+0000 7f6c9bed0640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c94134910 con 0x7f6c94143d10 2024-03-29T11:16:30.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.950+0000 7f6c9bed0640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:30.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.950+0000 7f6c9bed0640 10 monclient(hunting): authenticate will time out at 1203.147197s 2024-03-29T11:16:30.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.954+0000 7f6c9a46d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6c941419e0 0x7f6c941defa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:55683/0 (socket says 172.21.15.157:55683) 2024-03-29T11:16:30.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.954+0000 7f6c9a46d640 1 -- 172.21.15.157:0/3115276919 learned_addr learned my addr 172.21.15.157:0/3115276919 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:30.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 client.?.objecter ms_handle_connect 0x7f6c941419e0 2024-03-29T11:16:30.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:30.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 client.?.objecter ms_handle_connect 0x7f6c94143d10 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273177677 0 0) 0x7f6c94134910 con 0x7f6c94143d10 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c78004eb0 con 0x7f6c94143d10 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971620688 0 0) 0x7f6c94154b90 con 0x7f6c941419e0 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:16:30.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.955+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c94134910 con 0x7f6c941419e0 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1666331278 0 0) 0x7f6c78004eb0 con 0x7f6c94143d10 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c94154b90 con 0x7f6c94143d10 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6c84001b70 con 0x7f6c94143d10 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 10 monclient(hunting): dump: 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:30.938 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 10 client.?.objecter ms_handle_connect 0x7f6c940b1f60 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1392172453 0 0) 0x7f6c94134910 con 0x7f6c941419e0 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c78004eb0 con 0x7f6c941419e0 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.956+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430919510 0 0) 0x7f6c941548b0 con 0x7f6c940b1f60 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:16:30.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c94134910 con 0x7f6c940b1f60 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 642381615 0 0) 0x7f6c94154b90 con 0x7f6c94143d10 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 >> v1:172.21.15.195:6789/0 conn(0x7f6c941419e0 legacy=0x7f6c941defa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 >> v1:172.21.15.157:6789/0 conn(0x7f6c940b1f60 legacy=0x7f6c94153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 1 monclient: found mon.c 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c941345c0 con 0x7f6c94143d10 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.957+0000 7f6c8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c84001010 con 0x7f6c94143d10 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c9bed0640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c9bed0640 10 monclient: _renew_subs 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c9bed0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:30.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c9bed0640 1 -- 172.21.15.157:0/3115276919 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c94155390 con 0x7f6c94143d10 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c9bed0640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c9bed0640 10 monclient: _renew_subs 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c9bed0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c9bed0640 1 -- 172.21.15.157:0/3115276919 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c94133c90 con 0x7f6c94143d10 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c9bed0640 1 librados: init done 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c8bfff640 4 set_mon_vals no callback set 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6c84004d90 con 0x7f6c94143d10 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.958+0000 7f6c8a7fc640 10 monclient: dump: 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:30.941 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:30.942 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:30.942 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:30.942 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:30.942 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:30.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.960+0000 7f6c9bed0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:30.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.960+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f6c8401ad60 con 0x7f6c94143d10 2024-03-29T11:16:30.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.960+0000 7f6c9bed0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:30.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.960+0000 7f6c9bed0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:30.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.960+0000 7f6c9bed0640 1 -- 172.21.15.157:0/3115276919 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c005080 con 0x7f6c94143d10 2024-03-29T11:16:30.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.961+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f6c84084a90 con 0x7f6c94143d10 2024-03-29T11:16:30.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.961+0000 7f6c8a7fc640 10 client.4427.objecter ms_dispatch 0x7f6c94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:30.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.961+0000 7f6c8a7fc640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:30.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.961+0000 7f6c8a7fc640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:30.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.961+0000 7f6c8a7fc640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T11:16:30.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.965+0000 7f6c99c6c640 10 monclient: get_auth_request con 0x7f6c7806dfc0 auth_method 0 2024-03-29T11:16:30.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.966+0000 7f6c8a7fc640 10 client.4427.objecter ms_handle_connect 0x7f6c7806dfc0 2024-03-29T11:16:30.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.969+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c84050370 con 0x7f6c94143d10 2024-03-29T11:16:30.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.969+0000 7f6c8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:30.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:30.969+0000 7f6c8a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:31.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.312+0000 7f6c9bed0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:31.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.312+0000 7f6c9bed0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:31.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.312+0000 7f6c9bed0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:16:31.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.312+0000 7f6c9bed0640 1 -- 172.21.15.157:0/3115276919 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c7c005330 con 0x7f6c94143d10 2024-03-29T11:16:31.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.313+0000 7f6c8a7fc640 1 -- 172.21.15.157:0/3115276919 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f6c840522a0 con 0x7f6c94143d10 2024-03-29T11:16:31.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.313+0000 7f6c8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:31.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.313+0000 7f6c8a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:31.291 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:16:31.291 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:31.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.317+0000 7f6c9bed0640 10 librados: watch_flush enter 2024-03-29T11:16:31.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.317+0000 7f6c9bed0640 10 librados: watch_flush exit 2024-03-29T11:16:31.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.317+0000 7f6c9bed0640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T11:16:31.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.317+0000 7f6c9bed0640 10 client.4427.objecter successfully canceled tick 2024-03-29T11:16:31.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.317+0000 7f6c9bed0640 1 -- 172.21.15.157:0/3115276919 >> v1:172.21.15.157:6816/1747058247 conn(0x7f6c7806dfc0 legacy=0x7f6c78070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:31.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.317+0000 7f6c9bed0640 10 monclient: shutdown 2024-03-29T11:16:31.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.317+0000 7f6c9bed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:31.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.317+0000 7f6c9bed0640 1 -- 172.21.15.157:0/3115276919 >> v1:172.21.15.157:6790/0 conn(0x7f6c94143d10 legacy=0x7f6c94070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:31.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.317+0000 7f6c9bed0640 1 -- 172.21.15.157:0/3115276919 shutdown_connections 2024-03-29T11:16:31.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.317+0000 7f6c9bed0640 1 -- 172.21.15.157:0/3115276919 >> 172.21.15.157:0/3115276919 conn(0x7f6c94066a00 msgr2=0x7f6c94066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:31.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.318+0000 7f6c9bed0640 1 -- 172.21.15.157:0/3115276919 shutdown_connections 2024-03-29T11:16:31.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.318+0000 7f6c9bed0640 1 -- 172.21.15.157:0/3115276919 wait complete. 2024-03-29T11:16:31.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:31.318+0000 7f6c9bed0640 1 librados: shutdown 2024-03-29T11:16:31.314 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:38.121 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:38.279 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.300+0000 7ff3148c1640 10 monclient: get_monmap_and_config 2024-03-29T11:16:38.279 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.300+0000 7ff3148c1640 10 monclient: build_initial_monmap 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.300+0000 7ff3148c1640 10 monclient: monmap: 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:16:38.302025+0000 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:16:38.302025+0000 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:38.280 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.302+0000 7ff3148c1640 1 Processor -- start 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.302+0000 7ff3148c1640 1 -- start start 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.302+0000 7ff3148c1640 10 monclient: init 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.303+0000 7ff3148c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.303+0000 7ff3148c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.303+0000 7ff3148c1640 10 monclient(hunting): picked mon.noname-c con 0x7ff3100b4100 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.303+0000 7ff3148c1640 10 monclient(hunting): picked mon.noname-b con 0x7ff31013ee90 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.303+0000 7ff3148c1640 10 monclient(hunting): picked mon.noname-a con 0x7ff3100af900 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.303+0000 7ff3148c1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3100afd10 con 0x7ff3100af900 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.303+0000 7ff3148c1640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3100af6f0 con 0x7ff3100b4100 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.303+0000 7ff3148c1640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3100aac80 con 0x7ff31013ee90 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.303+0000 7ff3148c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:38.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.303+0000 7ff3148c1640 10 monclient(hunting): authenticate will time out at 1210.500789s 2024-03-29T11:16:38.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.308+0000 7ff30dd9b640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ff31013ee90 0x7ff31013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:59215/0 (socket says 172.21.15.157:59215) 2024-03-29T11:16:38.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.308+0000 7ff30ed9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff3100af900 0x7ff31014be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:54377/0 (socket says 172.21.15.157:54377) 2024-03-29T11:16:38.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.308+0000 7ff30dd9b640 1 -- 172.21.15.157:0/4203134181 learned_addr learned my addr 172.21.15.157:0/4203134181 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:38.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238586532 0 0) 0x7ff3100afd10 con 0x7ff3100af900 2024-03-29T11:16:38.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:16:38.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:38.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:38.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:16:38.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f8003520 con 0x7ff3100af900 2024-03-29T11:16:38.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559233811 0 0) 0x7ff3100aac80 con 0x7ff31013ee90 2024-03-29T11:16:38.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:16:38.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:38.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:38.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:16:38.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f8003840 con 0x7ff31013ee90 2024-03-29T11:16:38.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4162850303 0 0) 0x7ff2f8003520 con 0x7ff3100af900 2024-03-29T11:16:38.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 >> v1:172.21.15.157:6790/0 conn(0x7ff31013ee90 legacy=0x7ff31013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:38.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 >> v1:172.21.15.195:6789/0 conn(0x7ff3100b4100 legacy=0x7ff3100adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:38.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:16:38.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 1 monclient: found mon.noname-a 2024-03-29T11:16:38.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3100aaf60 con 0x7ff3100af900 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient: _finish_auth 0 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff308001970 con 0x7ff3100af900 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff30d59a640 10 monclient: dump: 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:38.296 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff3148c1640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.309+0000 7ff3148c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.310+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff308001ad0 con 0x7ff3100af900 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.310+0000 7ff30d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.310+0000 7ff30d59a640 1 -- 172.21.15.157:0/4203134181 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff308001f20 con 0x7ff3100af900 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.310+0000 7ff30d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.310+0000 7ff30d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.310+0000 7ff30d59a640 10 monclient: dump: 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:38.297 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.311+0000 7ff3148c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.311+0000 7ff3148c1640 4 set_mon_vals no callback set 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.312+0000 7ff3148c1640 10 monclient: shutdown 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.312+0000 7ff3148c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.312+0000 7ff3148c1640 1 -- 172.21.15.157:0/4203134181 >> v1:172.21.15.157:6789/0 conn(0x7ff3100af900 legacy=0x7ff31014be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.312+0000 7ff3148c1640 1 -- 172.21.15.157:0/4203134181 shutdown_connections 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.312+0000 7ff3148c1640 1 -- 172.21.15.157:0/4203134181 >> 172.21.15.157:0/4203134181 conn(0x7ff3100b35e0 msgr2=0x7ff3100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:38.298 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.314+0000 7ff3148c1640 1 -- 172.21.15.157:0/4203134181 shutdown_connections 2024-03-29T11:16:38.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.314+0000 7ff3148c1640 1 -- 172.21.15.157:0/4203134181 wait complete. 2024-03-29T11:16:38.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.315+0000 7ff3148c1640 10 monclient: build_initial_monmap 2024-03-29T11:16:38.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.315+0000 7ff3148c1640 10 monclient: monmap: 2024-03-29T11:16:38.300 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:38.300 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:38.300 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:16:38.300 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:16:38.300 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:38.300 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:38.300 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:38.300 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.316+0000 7ff3148c1640 1 librados: starting msgr at 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.316+0000 7ff3148c1640 1 librados: starting objecter 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.316+0000 7ff3148c1640 1 Processor -- start 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.316+0000 7ff3148c1640 1 -- start start 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.316+0000 7ff3148c1640 1 librados: setting wanted keys 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.316+0000 7ff3148c1640 1 librados: calling monclient init 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.316+0000 7ff3148c1640 10 monclient: init 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.317+0000 7ff3148c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:38.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.317+0000 7ff3148c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:38.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.319+0000 7ff3148c1640 10 monclient(hunting): picked mon.noname-b con 0x7ff3100af900 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:38.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.320+0000 7ff3148c1640 10 monclient(hunting): picked mon.noname-c con 0x7ff3100b4100 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:38.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.321+0000 7ff30e59c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff3100af900 0x7ff31013aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:45463/0 (socket says 172.21.15.157:45463) 2024-03-29T11:16:38.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.322+0000 7ff30dd9b640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ff3100b4100 0x7ff31014f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:59283/0 (socket says 172.21.15.157:59283) 2024-03-29T11:16:38.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.322+0000 7ff3148c1640 10 monclient(hunting): picked mon.noname-a con 0x7ff31013ee90 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:38.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.322+0000 7ff3148c1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31006ff00 con 0x7ff31013ee90 2024-03-29T11:16:38.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.322+0000 7ff3148c1640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff310071b00 con 0x7ff3100af900 2024-03-29T11:16:38.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.322+0000 7ff3148c1640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff310071d70 con 0x7ff3100b4100 2024-03-29T11:16:38.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.322+0000 7ff3148c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:38.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.322+0000 7ff3148c1640 10 monclient(hunting): authenticate will time out at 1210.519826s 2024-03-29T11:16:38.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.325+0000 7ff30e59c640 1 -- 172.21.15.157:0/1566336509 learned_addr learned my addr 172.21.15.157:0/1566336509 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:38.307 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.325+0000 7ff2feffd640 10 client.?.objecter ms_handle_connect 0x7ff3100af900 2024-03-29T11:16:38.307 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.325+0000 7ff2feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:38.307 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.327+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566687449 0 0) 0x7ff310071b00 con 0x7ff3100af900 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.327+0000 7ff2feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.327+0000 7ff2feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.327+0000 7ff2feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.327+0000 7ff2feffd640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.327+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f0004eb0 con 0x7ff3100af900 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.328+0000 7ff2feffd640 10 client.?.objecter ms_handle_connect 0x7ff3100b4100 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.328+0000 7ff2feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.328+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3517021743 0 0) 0x7ff2f0004eb0 con 0x7ff3100af900 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.328+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff310071b00 con 0x7ff3100af900 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.328+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff304001a30 con 0x7ff3100af900 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.328+0000 7ff2feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.328+0000 7ff2feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:38.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.328+0000 7ff2feffd640 10 monclient(hunting): dump: 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.328+0000 7ff2feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:38.309 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.329+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290919777 0 0) 0x7ff310071d70 con 0x7ff3100b4100 2024-03-29T11:16:38.310 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.329+0000 7ff2feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:38.310 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.329+0000 7ff2feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:38.310 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.329+0000 7ff2feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:38.310 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.329+0000 7ff2feffd640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:16:38.310 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.329+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f0004eb0 con 0x7ff3100b4100 2024-03-29T11:16:38.313 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.332+0000 7ff2feffd640 10 client.?.objecter ms_handle_connect 0x7ff31013ee90 2024-03-29T11:16:38.313 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.332+0000 7ff2feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:38.313 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.332+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1249912948 0 0) 0x7ff310071b00 con 0x7ff3100af900 2024-03-29T11:16:38.313 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.332+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 >> v1:172.21.15.157:6790/0 conn(0x7ff3100b4100 legacy=0x7ff31014f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:38.313 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 >> v1:172.21.15.157:6789/0 conn(0x7ff31013ee90 legacy=0x7ff310070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:38.313 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff2feffd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:38.313 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff2feffd640 1 monclient: found mon.b 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff2feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff2feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff310072050 con 0x7ff3100af900 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff2feffd640 10 monclient: _finish_auth 0 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff2feffd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff2feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff304001cf0 con 0x7ff3100af900 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff3148c1640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff3148c1640 10 monclient: _renew_subs 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff3148c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff3148c1640 1 -- 172.21.15.157:0/1566336509 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff310071180 con 0x7ff3100af900 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff3148c1640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff3148c1640 10 monclient: _renew_subs 2024-03-29T11:16:38.314 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff3148c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff3148c1640 1 -- 172.21.15.157:0/1566336509 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3101e7d30 con 0x7ff3100af900 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.333+0000 7ff3148c1640 1 librados: init done 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.335+0000 7ff2feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.335+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff304003a60 con 0x7ff3100af900 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.335+0000 7ff30cd99640 4 set_mon_vals no callback set 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.335+0000 7ff2feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.335+0000 7ff2feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.335+0000 7ff2feffd640 10 monclient: dump: 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:38.315 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:38.316 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:38.316 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:38.316 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:38.316 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:38.316 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:38.316 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:38.316 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:38.317 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.336+0000 7ff2fcff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:38.317 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.336+0000 7ff2fcff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:38.317 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.336+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7ff304003c80 con 0x7ff3100af900 2024-03-29T11:16:38.317 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.336+0000 7ff2fcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:38.317 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.336+0000 7ff2fcff9640 1 -- 172.21.15.157:0/1566336509 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f4005080 con 0x7ff3100af900 2024-03-29T11:16:38.317 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.337+0000 7ff30dd9b640 10 monclient: get_auth_request con 0x7ff2f006dd30 auth_method 0 2024-03-29T11:16:38.317 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.338+0000 7ff2feffd640 10 client.4399.objecter ms_handle_connect 0x7ff2f006dd30 2024-03-29T11:16:38.318 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.338+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7ff304084ba0 con 0x7ff3100af900 2024-03-29T11:16:38.318 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.338+0000 7ff2feffd640 10 client.4399.objecter ms_dispatch 0x7ff310000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:38.318 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.338+0000 7ff2feffd640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:38.318 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.338+0000 7ff2feffd640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:38.318 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.338+0000 7ff2feffd640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-29T11:16:38.322 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.344+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff304050480 con 0x7ff3100af900 2024-03-29T11:16:38.322 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.345+0000 7ff2feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:38.322 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.345+0000 7ff2feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:38.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.675+0000 7ff2fcff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:38.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.675+0000 7ff2fcff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:38.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.675+0000 7ff2fcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:38.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.675+0000 7ff2fcff9640 1 -- 172.21.15.157:0/1566336509 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2f4005330 con 0x7ff3100af900 2024-03-29T11:16:38.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.678+0000 7ff2feffd640 1 -- 172.21.15.157:0/1566336509 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7ff3040523b0 con 0x7ff3100af900 2024-03-29T11:16:38.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.678+0000 7ff2feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:38.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.678+0000 7ff2feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:38.656 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:16:38.656 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:38.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.682+0000 7ff2fcff9640 10 librados: watch_flush enter 2024-03-29T11:16:38.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.682+0000 7ff2fcff9640 10 librados: watch_flush exit 2024-03-29T11:16:38.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.682+0000 7ff2fcff9640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-29T11:16:38.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.682+0000 7ff2fcff9640 10 client.4399.objecter successfully canceled tick 2024-03-29T11:16:38.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.682+0000 7ff2fcff9640 1 -- 172.21.15.157:0/1566336509 >> v1:172.21.15.157:6816/1747058247 conn(0x7ff2f006dd30 legacy=0x7ff2f00701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:38.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.682+0000 7ff2fcff9640 10 monclient: shutdown 2024-03-29T11:16:38.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.682+0000 7ff2fcff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:38.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.682+0000 7ff2fcff9640 1 -- 172.21.15.157:0/1566336509 >> v1:172.21.15.195:6789/0 conn(0x7ff3100af900 legacy=0x7ff31013aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:38.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.683+0000 7ff2fcff9640 1 -- 172.21.15.157:0/1566336509 shutdown_connections 2024-03-29T11:16:38.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.684+0000 7ff2fcff9640 1 -- 172.21.15.157:0/1566336509 >> 172.21.15.157:0/1566336509 conn(0x7ff3100b35e0 msgr2=0x7ff3100b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:38.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.684+0000 7ff2fcff9640 1 -- 172.21.15.157:0/1566336509 shutdown_connections 2024-03-29T11:16:38.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.684+0000 7ff2fcff9640 1 -- 172.21.15.157:0/1566336509 wait complete. 2024-03-29T11:16:38.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:38.686+0000 7ff2fcff9640 1 librados: shutdown 2024-03-29T11:16:38.680 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:45.486 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:45.637 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.659+0000 7f201d4f2640 10 monclient: get_monmap_and_config 2024-03-29T11:16:45.637 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.659+0000 7f201d4f2640 10 monclient: build_initial_monmap 2024-03-29T11:16:45.637 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.659+0000 7f201d4f2640 10 monclient: monmap: 2024-03-29T11:16:45.637 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:45.637 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:45.637 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:16:45.660420+0000 2024-03-29T11:16:45.638 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:16:45.660420+0000 2024-03-29T11:16:45.638 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:45.638 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:45.638 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:45.638 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:16:45.638 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:16:45.639 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:45.639 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.660+0000 7f201d4f2640 1 Processor -- start 2024-03-29T11:16:45.639 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.660+0000 7f201d4f2640 1 -- start start 2024-03-29T11:16:45.639 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.660+0000 7f201d4f2640 10 monclient: init 2024-03-29T11:16:45.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.661+0000 7f201d4f2640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:45.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.661+0000 7f201d4f2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:45.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.661+0000 7f201d4f2640 10 monclient(hunting): picked mon.noname-a con 0x7f20100389e0 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:45.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.661+0000 7f201d4f2640 10 monclient(hunting): picked mon.noname-c con 0x7f2010038dc0 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:45.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.661+0000 7f201d4f2640 10 monclient(hunting): picked mon.noname-b con 0x7f2010030660 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:45.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.661+0000 7f201d4f2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20100c54c0 con 0x7f20100389e0 2024-03-29T11:16:45.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.661+0000 7f201d4f2640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2010036430 con 0x7f2010038dc0 2024-03-29T11:16:45.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.661+0000 7f201d4f2640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2010039840 con 0x7f2010030660 2024-03-29T11:16:45.641 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.661+0000 7f201d4f2640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:45.641 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.661+0000 7f201d4f2640 10 monclient(hunting): authenticate will time out at 1217.858316s 2024-03-29T11:16:45.644 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.666+0000 7f2017fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f20100389e0 0x7f2010033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:55677/0 (socket says 172.21.15.157:55677) 2024-03-29T11:16:45.644 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.667+0000 7f2017fff640 1 -- 172.21.15.157:0/259269106 learned_addr learned my addr 172.21.15.157:0/259269106 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 1 -- 172.21.15.157:0/259269106 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403064064 0 0) 0x7f20100c54c0 con 0x7f20100389e0 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 1 -- 172.21.15.157:0/259269106 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2008003520 con 0x7f20100389e0 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 1 -- 172.21.15.157:0/259269106 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007471706 0 0) 0x7f2008003520 con 0x7f20100389e0 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 1 -- 172.21.15.157:0/259269106 >> v1:172.21.15.157:6790/0 conn(0x7f2010030660 legacy=0x7f20100d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 1 -- 172.21.15.157:0/259269106 >> v1:172.21.15.195:6789/0 conn(0x7f2010038dc0 legacy=0x7f20100cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 1 monclient: found mon.noname-a 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:45.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 1 -- 172.21.15.157:0/259269106 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2010035690 con 0x7f20100389e0 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 10 monclient: _finish_auth 0 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 1 -- 172.21.15.157:0/259269106 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f200c001690 con 0x7f20100389e0 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.668+0000 7f2016ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f2016ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f2016ffd640 10 monclient: dump: 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:45.649 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f2016ffd640 1 -- 172.21.15.157:0/259269106 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f200c001860 con 0x7f20100389e0 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f2016ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f2016ffd640 1 -- 172.21.15.157:0/259269106 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f200c001cb0 con 0x7f20100389e0 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f2016ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f2016ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f2016ffd640 10 monclient: dump: 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:45.650 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f201d4f2640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f201d4f2640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.669+0000 7f201d4f2640 4 set_mon_vals no callback set 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.670+0000 7f201d4f2640 10 monclient: shutdown 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.670+0000 7f201d4f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.670+0000 7f201d4f2640 1 -- 172.21.15.157:0/259269106 >> v1:172.21.15.157:6789/0 conn(0x7f20100389e0 legacy=0x7f2010033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.670+0000 7f201d4f2640 1 -- 172.21.15.157:0/259269106 shutdown_connections 2024-03-29T11:16:45.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.670+0000 7f201d4f2640 1 -- 172.21.15.157:0/259269106 >> 172.21.15.157:0/259269106 conn(0x7f2010005c00 msgr2=0x7f2010006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:45.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.672+0000 7f201d4f2640 1 -- 172.21.15.157:0/259269106 shutdown_connections 2024-03-29T11:16:45.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.673+0000 7f201d4f2640 1 -- 172.21.15.157:0/259269106 wait complete. 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.674+0000 7f201d4f2640 10 monclient: build_initial_monmap 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.674+0000 7f201d4f2640 10 monclient: monmap: 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:16:45.653 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:16:45.654 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:45.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.675+0000 7f201d4f2640 1 librados: starting msgr at 2024-03-29T11:16:45.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.675+0000 7f201d4f2640 1 librados: starting objecter 2024-03-29T11:16:45.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.675+0000 7f201d4f2640 1 Processor -- start 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.675+0000 7f201d4f2640 1 -- start start 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.675+0000 7f201d4f2640 1 librados: setting wanted keys 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.675+0000 7f201d4f2640 1 librados: calling monclient init 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.675+0000 7f201d4f2640 10 monclient: init 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.676+0000 7f201d4f2640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.676+0000 7f201d4f2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.676+0000 7f201d4f2640 10 monclient(hunting): picked mon.noname-b con 0x7f2010030660 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.676+0000 7f201d4f2640 10 monclient(hunting): picked mon.noname-a con 0x7f20100389e0 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.676+0000 7f201d4f2640 10 monclient(hunting): picked mon.noname-c con 0x7f201000cb60 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.676+0000 7f201d4f2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2010030040 con 0x7f20100389e0 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.676+0000 7f201d4f2640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201002f270 con 0x7f2010030660 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.676+0000 7f201d4f2640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201000d600 con 0x7f201000cb60 2024-03-29T11:16:45.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.676+0000 7f201d4f2640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:45.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.676+0000 7f201d4f2640 10 monclient(hunting): authenticate will time out at 1217.873434s 2024-03-29T11:16:45.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.682+0000 7f20177fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f20100389e0 0x7f201000be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:55967/0 (socket says 172.21.15.157:55967) 2024-03-29T11:16:45.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.683+0000 7f20177fe640 1 -- 172.21.15.157:0/1995910597 learned_addr learned my addr 172.21.15.157:0/1995910597 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:45.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.684+0000 7f2014ff9640 10 client.?.objecter ms_handle_connect 0x7f20100389e0 2024-03-29T11:16:45.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.684+0000 7f2014ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:45.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.684+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968059520 0 0) 0x7f2010030040 con 0x7f20100389e0 2024-03-29T11:16:45.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.684+0000 7f2014ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:45.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.684+0000 7f2014ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:45.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.684+0000 7f2014ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:45.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.684+0000 7f2014ff9640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:16:45.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.684+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ffc004eb0 con 0x7f20100389e0 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.689+0000 7f2014ff9640 10 client.?.objecter ms_handle_connect 0x7f2010030660 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.689+0000 7f2014ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 client.?.objecter ms_handle_connect 0x7f201000cb60 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281070405 0 0) 0x7f201002f270 con 0x7f2010030660 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2010030040 con 0x7f2010030660 2024-03-29T11:16:45.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3216018961 0 0) 0x7f1ffc004eb0 con 0x7f20100389e0 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f201002f270 con 0x7f20100389e0 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992141944 0 0) 0x7f201000d600 con 0x7f201000cb60 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ffc004eb0 con 0x7f201000cb60 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2004001840 con 0x7f20100389e0 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): dump: 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:45.672 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.690+0000 7f2014ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.691+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2871627313 0 0) 0x7f201002f270 con 0x7f20100389e0 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.691+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 >> v1:172.21.15.157:6790/0 conn(0x7f201000cb60 legacy=0x7f201002dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.691+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 >> v1:172.21.15.195:6789/0 conn(0x7f2010030660 legacy=0x7f2010164610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.691+0000 7f2014ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.691+0000 7f2014ff9640 1 monclient: found mon.a 2024-03-29T11:16:45.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.691+0000 7f2014ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:45.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.691+0000 7f2014ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:16:45.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201000d790 con 0x7f20100389e0 2024-03-29T11:16:45.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f2014ff9640 10 monclient: _finish_auth 0 2024-03-29T11:16:45.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f2014ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:16:45.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f2014ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:45.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f201d4f2640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:16:45.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f201d4f2640 10 monclient: _renew_subs 2024-03-29T11:16:45.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f201d4f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:16:45.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f201d4f2640 1 -- 172.21.15.157:0/1995910597 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201000c560 con 0x7f20100389e0 2024-03-29T11:16:45.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f201d4f2640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:45.675 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f201d4f2640 10 monclient: _renew_subs 2024-03-29T11:16:45.675 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f201d4f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:16:45.675 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f201d4f2640 1 -- 172.21.15.157:0/1995910597 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f201002e2d0 con 0x7f20100389e0 2024-03-29T11:16:45.675 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.692+0000 7f201d4f2640 1 librados: init done 2024-03-29T11:16:45.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.694+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2004001d30 con 0x7f20100389e0 2024-03-29T11:16:45.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.694+0000 7f2014ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:45.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.694+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f20040037b0 con 0x7f20100389e0 2024-03-29T11:16:45.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.694+0000 7f2014ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:45.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.694+0000 7f2014ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:45.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.694+0000 7f20167fc640 4 set_mon_vals no callback set 2024-03-29T11:16:45.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.695+0000 7f2014ff9640 10 monclient: dump: 2024-03-29T11:16:45.677 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:45.677 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:45.677 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:45.677 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:45.678 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:45.678 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:45.678 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:45.678 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:45.678 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:45.678 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:45.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.695+0000 7f201d4f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:45.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.695+0000 7f201d4f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:45.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.696+0000 7f201d4f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:16:45.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.696+0000 7f201d4f2640 1 -- 172.21.15.157:0/1995910597 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff8005080 con 0x7f20100389e0 2024-03-29T11:16:45.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.696+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f2004003910 con 0x7f20100389e0 2024-03-29T11:16:45.679 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.697+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f20040848f0 con 0x7f20100389e0 2024-03-29T11:16:45.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.697+0000 7f2014ff9640 10 client.4434.objecter ms_dispatch 0x7f2018000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:45.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.697+0000 7f2014ff9640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:45.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.697+0000 7f2014ff9640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:45.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.697+0000 7f2014ff9640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T11:16:45.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.698+0000 7f2017fff640 10 monclient: get_auth_request con 0x7f1ffc06ddf0 auth_method 0 2024-03-29T11:16:45.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.698+0000 7f2014ff9640 10 client.4434.objecter ms_handle_connect 0x7f1ffc06ddf0 2024-03-29T11:16:45.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.706+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2004053d90 con 0x7f20100389e0 2024-03-29T11:16:45.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.706+0000 7f2014ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:45.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:45.706+0000 7f2014ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:46.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.040+0000 7f201d4f2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:46.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.040+0000 7f201d4f2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:46.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.040+0000 7f201d4f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:16:46.018 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.040+0000 7f201d4f2640 1 -- 172.21.15.157:0/1995910597 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ff8005330 con 0x7f20100389e0 2024-03-29T11:16:46.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.042+0000 7f2014ff9640 1 -- 172.21.15.157:0/1995910597 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f2004052100 con 0x7f20100389e0 2024-03-29T11:16:46.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.042+0000 7f2014ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:46.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.042+0000 7f2014ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:46.020 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:16:46.020 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:46.024 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.045+0000 7f201d4f2640 10 librados: watch_flush enter 2024-03-29T11:16:46.024 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.047+0000 7f201d4f2640 10 librados: watch_flush exit 2024-03-29T11:16:46.024 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.047+0000 7f201d4f2640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T11:16:46.025 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.047+0000 7f201d4f2640 10 client.4434.objecter successfully canceled tick 2024-03-29T11:16:46.025 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.048+0000 7f201d4f2640 1 -- 172.21.15.157:0/1995910597 >> v1:172.21.15.157:6816/1747058247 conn(0x7f1ffc06ddf0 legacy=0x7f1ffc0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:46.025 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.048+0000 7f201d4f2640 10 monclient: shutdown 2024-03-29T11:16:46.025 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.048+0000 7f201d4f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:46.025 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.048+0000 7f201d4f2640 1 -- 172.21.15.157:0/1995910597 >> v1:172.21.15.157:6789/0 conn(0x7f20100389e0 legacy=0x7f201000be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:46.026 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.049+0000 7f201d4f2640 1 -- 172.21.15.157:0/1995910597 shutdown_connections 2024-03-29T11:16:46.026 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.049+0000 7f201d4f2640 1 -- 172.21.15.157:0/1995910597 >> 172.21.15.157:0/1995910597 conn(0x7f2010005c00 msgr2=0x7f2010005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:46.026 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.049+0000 7f201d4f2640 1 -- 172.21.15.157:0/1995910597 shutdown_connections 2024-03-29T11:16:46.027 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.049+0000 7f201d4f2640 1 -- 172.21.15.157:0/1995910597 wait complete. 2024-03-29T11:16:46.027 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:46.049+0000 7f201d4f2640 1 librados: shutdown 2024-03-29T11:16:46.043 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:52.851 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.023+0000 7ff235594640 10 monclient: get_monmap_and_config 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.023+0000 7ff235594640 10 monclient: build_initial_monmap 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.023+0000 7ff235594640 10 monclient: monmap: 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:16:53.024270+0000 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:16:53.024270+0000 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:16:53.003 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:16:53.004 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:53.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.024+0000 7ff235594640 1 Processor -- start 2024-03-29T11:16:53.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.024+0000 7ff235594640 1 -- start start 2024-03-29T11:16:53.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.024+0000 7ff235594640 10 monclient: init 2024-03-29T11:16:53.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.025+0000 7ff235594640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:53.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.025+0000 7ff235594640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:53.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.025+0000 7ff235594640 10 monclient(hunting): picked mon.noname-a con 0x7ff2300aaee0 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:53.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.026+0000 7ff235594640 10 monclient(hunting): picked mon.noname-b con 0x7ff2300ae460 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:53.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.026+0000 7ff235594640 10 monclient(hunting): picked mon.noname-c con 0x7ff2300aeff0 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:53.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.026+0000 7ff235594640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2300abb60 con 0x7ff2300aaee0 2024-03-29T11:16:53.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.026+0000 7ff235594640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2300abd90 con 0x7ff2300aeff0 2024-03-29T11:16:53.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.026+0000 7ff235594640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2300ac070 con 0x7ff2300ae460 2024-03-29T11:16:53.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.026+0000 7ff235594640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:53.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.026+0000 7ff235594640 10 monclient(hunting): authenticate will time out at 1225.223474s 2024-03-29T11:16:53.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.028+0000 7ff22f7fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ff2300ae460 0x7ff2300ae8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:46946/0 (socket says 172.21.15.157:46946) 2024-03-29T11:16:53.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.028+0000 7ff22f7fe640 1 -- 172.21.15.157:0/748718675 learned_addr learned my addr 172.21.15.157:0/748718675 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:53.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462182174 0 0) 0x7ff2300ac070 con 0x7ff2300ae460 2024-03-29T11:16:53.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:53.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:53.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:53.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:16:53.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff218003520 con 0x7ff2300ae460 2024-03-29T11:16:53.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339883327 0 0) 0x7ff2300abb60 con 0x7ff2300aaee0 2024-03-29T11:16:53.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:53.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:53.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2300ac070 con 0x7ff2300aaee0 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1036828725 0 0) 0x7ff218003520 con 0x7ff2300ae460 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 >> v1:172.21.15.195:6789/0 conn(0x7ff2300aeff0 legacy=0x7ff2300af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 >> v1:172.21.15.157:6789/0 conn(0x7ff2300aaee0 legacy=0x7ff2300acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 1 monclient: found mon.noname-b 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2300ac350 con 0x7ff2300ae460 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient: _finish_auth 0 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:53.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff228005010 con 0x7ff2300ae460 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.032+0000 7ff22effd640 10 monclient: dump: 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:53.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.033+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff228005230 con 0x7ff2300ae460 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.033+0000 7ff22effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.033+0000 7ff22effd640 1 -- 172.21.15.157:0/748718675 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff228001790 con 0x7ff2300ae460 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.033+0000 7ff22effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.033+0000 7ff22effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.033+0000 7ff22effd640 10 monclient: dump: 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:53.013 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:53.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.034+0000 7ff235594640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:16:53.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.034+0000 7ff235594640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:53.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.035+0000 7ff235594640 4 set_mon_vals no callback set 2024-03-29T11:16:53.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.035+0000 7ff235594640 10 monclient: shutdown 2024-03-29T11:16:53.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.035+0000 7ff235594640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:53.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.035+0000 7ff235594640 1 -- 172.21.15.157:0/748718675 >> v1:172.21.15.157:6790/0 conn(0x7ff2300ae460 legacy=0x7ff2300ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:53.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.036+0000 7ff235594640 1 -- 172.21.15.157:0/748718675 shutdown_connections 2024-03-29T11:16:53.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.036+0000 7ff235594640 1 -- 172.21.15.157:0/748718675 >> 172.21.15.157:0/748718675 conn(0x7ff2300b3a30 msgr2=0x7ff2300b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:53.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.036+0000 7ff235594640 1 -- 172.21.15.157:0/748718675 shutdown_connections 2024-03-29T11:16:53.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.036+0000 7ff235594640 1 -- 172.21.15.157:0/748718675 wait complete. 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.037+0000 7ff235594640 10 monclient: build_initial_monmap 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.037+0000 7ff235594640 10 monclient: monmap: 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.037+0000 7ff235594640 1 librados: starting msgr at 2024-03-29T11:16:53.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.037+0000 7ff235594640 1 librados: starting objecter 2024-03-29T11:16:53.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.037+0000 7ff235594640 1 Processor -- start 2024-03-29T11:16:53.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.038+0000 7ff235594640 1 -- start start 2024-03-29T11:16:53.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.038+0000 7ff235594640 1 librados: setting wanted keys 2024-03-29T11:16:53.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.038+0000 7ff235594640 1 librados: calling monclient init 2024-03-29T11:16:53.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.038+0000 7ff235594640 10 monclient: init 2024-03-29T11:16:53.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.038+0000 7ff235594640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:53.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.038+0000 7ff235594640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:53.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.040+0000 7ff235594640 10 monclient(hunting): picked mon.noname-b con 0x7ff2300aaee0 addr v1:172.21.15.195:6789/0 2024-03-29T11:16:53.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.041+0000 7ff235594640 10 monclient(hunting): picked mon.noname-c con 0x7ff2300ae460 addr v1:172.21.15.157:6790/0 2024-03-29T11:16:53.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.041+0000 7ff235594640 10 monclient(hunting): picked mon.noname-a con 0x7ff2300aeff0 addr v1:172.21.15.157:6789/0 2024-03-29T11:16:53.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.041+0000 7ff235594640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23006e370 con 0x7ff2300aeff0 2024-03-29T11:16:53.019 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.041+0000 7ff235594640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2301503d0 con 0x7ff2300aaee0 2024-03-29T11:16:53.020 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.041+0000 7ff235594640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff230088fe0 con 0x7ff2300ae460 2024-03-29T11:16:53.020 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.041+0000 7ff235594640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:53.020 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.041+0000 7ff235594640 10 monclient(hunting): authenticate will time out at 1225.238664s 2024-03-29T11:16:53.021 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.042+0000 7ff22ffff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff2300aaee0 0x7ff23014fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:60589/0 (socket says 172.21.15.157:60589) 2024-03-29T11:16:53.021 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.042+0000 7ff22ffff640 1 -- 172.21.15.157:0/2412013782 learned_addr learned my addr 172.21.15.157:0/2412013782 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:16:53.021 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.043+0000 7ff22cff9640 10 client.?.objecter ms_handle_connect 0x7ff2300aaee0 2024-03-29T11:16:53.021 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.043+0000 7ff22cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:53.021 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.043+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828416846 0 0) 0x7ff2301503d0 con 0x7ff2300aaee0 2024-03-29T11:16:53.021 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.043+0000 7ff22cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:53.021 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.043+0000 7ff22cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:53.021 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.043+0000 7ff22cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:53.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.043+0000 7ff22cff9640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:16:53.022 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.043+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff21c004eb0 con 0x7ff2300aaee0 2024-03-29T11:16:53.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.045+0000 7ff22cff9640 10 client.?.objecter ms_handle_connect 0x7ff2300ae460 2024-03-29T11:16:53.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.045+0000 7ff22cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:53.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.045+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2775055197 0 0) 0x7ff21c004eb0 con 0x7ff2300aaee0 2024-03-29T11:16:53.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.045+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2301503d0 con 0x7ff2300aaee0 2024-03-29T11:16:53.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.045+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff220001a10 con 0x7ff2300aaee0 2024-03-29T11:16:53.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.045+0000 7ff22cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:53.029 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.045+0000 7ff22cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.045+0000 7ff22cff9640 10 monclient(hunting): dump: 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.045+0000 7ff22cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:53.030 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263232715 0 0) 0x7ff230088fe0 con 0x7ff2300ae460 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff21c004eb0 con 0x7ff2300ae460 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 307599552 0 0) 0x7ff2301503d0 con 0x7ff2300aaee0 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 >> v1:172.21.15.157:6790/0 conn(0x7ff2300ae460 legacy=0x7ff2300881c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 >> v1:172.21.15.157:6789/0 conn(0x7ff2300aeff0 legacy=0x7ff2300888d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 1 monclient: found mon.b 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2301db2c0 con 0x7ff2300aaee0 2024-03-29T11:16:53.031 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 10 monclient: _finish_auth 0 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.046+0000 7ff22cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.047+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff220001b70 con 0x7ff2300aaee0 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.047+0000 7ff22cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.047+0000 7ff22dffb640 4 set_mon_vals no callback set 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.048+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff220003a60 con 0x7ff2300aaee0 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.048+0000 7ff22cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.048+0000 7ff22cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.048+0000 7ff22cff9640 10 monclient: dump: 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:16:53.032 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.048+0000 7ff235594640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.048+0000 7ff235594640 10 monclient: _renew_subs 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.048+0000 7ff235594640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.048+0000 7ff235594640 1 -- 172.21.15.157:0/2412013782 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23014f7a0 con 0x7ff2300aaee0 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.049+0000 7ff235594640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.049+0000 7ff235594640 10 monclient: _renew_subs 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.049+0000 7ff235594640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.049+0000 7ff235594640 1 -- 172.21.15.157:0/2412013782 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2300894d0 con 0x7ff2300aaee0 2024-03-29T11:16:53.033 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.049+0000 7ff235594640 1 librados: init done 2024-03-29T11:16:53.034 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.050+0000 7ff235594640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:53.034 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.050+0000 7ff235594640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:53.034 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.050+0000 7ff235594640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:53.034 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.050+0000 7ff235594640 1 -- 172.21.15.157:0/2412013782 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23004f730 con 0x7ff2300aaee0 2024-03-29T11:16:53.034 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.051+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7ff220003bc0 con 0x7ff2300aaee0 2024-03-29T11:16:53.035 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.052+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7ff220084b90 con 0x7ff2300aaee0 2024-03-29T11:16:53.035 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.052+0000 7ff22cff9640 10 client.4411.objecter ms_dispatch 0x7ff230000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:53.035 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.052+0000 7ff22cff9640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:53.035 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.052+0000 7ff22cff9640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:53.035 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.052+0000 7ff22cff9640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T11:16:53.035 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.052+0000 7ff22f7fe640 10 monclient: get_auth_request con 0x7ff21c06dc90 auth_method 0 2024-03-29T11:16:53.035 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.053+0000 7ff22cff9640 10 client.4411.objecter ms_handle_connect 0x7ff21c06dc90 2024-03-29T11:16:53.036 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.058+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff220050470 con 0x7ff2300aaee0 2024-03-29T11:16:53.036 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.058+0000 7ff22cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:53.036 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.058+0000 7ff22cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:53.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.379+0000 7ff235594640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:53.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.379+0000 7ff235594640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:53.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.379+0000 7ff235594640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:16:53.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.379+0000 7ff235594640 1 -- 172.21.15.157:0/2412013782 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff23014f9e0 con 0x7ff2300aaee0 2024-03-29T11:16:53.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.381+0000 7ff22cff9640 1 -- 172.21.15.157:0/2412013782 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7ff2200523a0 con 0x7ff2300aaee0 2024-03-29T11:16:53.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.381+0000 7ff22cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:53.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.381+0000 7ff22cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:53.359 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:16:53.359 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:53.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.384+0000 7ff235594640 10 librados: watch_flush enter 2024-03-29T11:16:53.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.385+0000 7ff235594640 10 librados: watch_flush exit 2024-03-29T11:16:53.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.385+0000 7ff235594640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T11:16:53.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.385+0000 7ff235594640 10 client.4411.objecter successfully canceled tick 2024-03-29T11:16:53.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.386+0000 7ff235594640 1 -- 172.21.15.157:0/2412013782 >> v1:172.21.15.157:6816/1747058247 conn(0x7ff21c06dc90 legacy=0x7ff21c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:53.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.386+0000 7ff235594640 10 monclient: shutdown 2024-03-29T11:16:53.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.386+0000 7ff235594640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:53.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.386+0000 7ff235594640 1 -- 172.21.15.157:0/2412013782 >> v1:172.21.15.195:6789/0 conn(0x7ff2300aaee0 legacy=0x7ff23014fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:53.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.388+0000 7ff235594640 1 -- 172.21.15.157:0/2412013782 shutdown_connections 2024-03-29T11:16:53.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.388+0000 7ff235594640 1 -- 172.21.15.157:0/2412013782 >> 172.21.15.157:0/2412013782 conn(0x7ff2300b3a30 msgr2=0x7ff2300b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:53.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.389+0000 7ff235594640 1 -- 172.21.15.157:0/2412013782 shutdown_connections 2024-03-29T11:16:53.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.389+0000 7ff235594640 1 -- 172.21.15.157:0/2412013782 wait complete. 2024-03-29T11:16:53.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:16:53.389+0000 7ff235594640 1 librados: shutdown 2024-03-29T11:16:53.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:00.190 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:00.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.371+0000 7f1fe4125640 10 monclient: get_monmap_and_config 2024-03-29T11:17:00.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.371+0000 7f1fe4125640 10 monclient: build_initial_monmap 2024-03-29T11:17:00.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.371+0000 7f1fe4125640 10 monclient: monmap: 2024-03-29T11:17:00.349 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:00.349 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:00.349 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:17:00.372669+0000 2024-03-29T11:17:00.349 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:17:00.372669+0000 2024-03-29T11:17:00.349 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:00.349 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:00.350 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:00.350 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:17:00.350 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:17:00.350 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:00.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.373+0000 7f1fe4125640 1 Processor -- start 2024-03-29T11:17:00.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.373+0000 7f1fe4125640 1 -- start start 2024-03-29T11:17:00.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.373+0000 7f1fe4125640 10 monclient: init 2024-03-29T11:17:00.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.374+0000 7f1fe4125640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:00.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.374+0000 7f1fe4125640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:00.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.374+0000 7f1fe4125640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd4038a00 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:00.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.374+0000 7f1fe4125640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd4038de0 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:00.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.374+0000 7f1fe4125640 10 monclient(hunting): picked mon.noname-a con 0x7f1fd4030680 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:00.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.374+0000 7f1fe4125640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd40c54e0 con 0x7f1fd4030680 2024-03-29T11:17:00.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.374+0000 7f1fe4125640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd4036450 con 0x7f1fd4038de0 2024-03-29T11:17:00.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.374+0000 7f1fe4125640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd4039860 con 0x7f1fd4038a00 2024-03-29T11:17:00.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.374+0000 7f1fe4125640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:00.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.374+0000 7f1fe4125640 10 monclient(hunting): authenticate will time out at 1232.571409s 2024-03-29T11:17:00.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.375+0000 7f1fe16c0640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f1fd4038de0 0x7f1fd40cadf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:44506/0 (socket says 172.21.15.157:44506) 2024-03-29T11:17:00.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.375+0000 7f1fe16c0640 1 -- 172.21.15.157:0/3275343202 learned_addr learned my addr 172.21.15.157:0/3275343202 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:00.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.377+0000 7f1fe0ebf640 1 -- 172.21.15.157:0/3275343202 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141028257 0 0) 0x7f1fd4036450 con 0x7f1fd4038de0 2024-03-29T11:17:00.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.377+0000 7f1fe0ebf640 10 monclient(hunting): _negotiate 2024-03-29T11:17:00.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.377+0000 7f1fe0ebf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:00.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.377+0000 7f1fe0ebf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:00.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.377+0000 7f1fe0ebf640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:17:00.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.377+0000 7f1fe0ebf640 1 -- 172.21.15.157:0/3275343202 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc8003520 con 0x7f1fd4038de0 2024-03-29T11:17:00.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 1 -- 172.21.15.157:0/3275343202 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1055533924 0 0) 0x7f1fc8003520 con 0x7f1fd4038de0 2024-03-29T11:17:00.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 1 -- 172.21.15.157:0/3275343202 >> v1:172.21.15.157:6790/0 conn(0x7f1fd4038a00 legacy=0x7f1fd4033540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 1 -- 172.21.15.157:0/3275343202 >> v1:172.21.15.157:6789/0 conn(0x7f1fd4030680 legacy=0x7f1fd40d3190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 10 monclient: _finish_hunting 0 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 1 monclient: found mon.noname-c 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 1 -- 172.21.15.157:0/3275343202 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd40356b0 con 0x7f1fd4038de0 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 10 monclient: _finish_auth 0 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 10 monclient: _check_auth_tickets 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 1 -- 172.21.15.157:0/3275343202 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1fd8001690 con 0x7f1fd4038de0 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe4125640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe4125640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:00.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe0ebf640 10 monclient: dump: 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:00.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.378+0000 7f1fe4125640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:00.359 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe0ebf640 1 -- 172.21.15.157:0/3275343202 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd8001920 con 0x7f1fd4038de0 2024-03-29T11:17:00.359 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe0ebf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:00.359 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe4125640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe4125640 4 set_mon_vals no callback set 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe0ebf640 1 -- 172.21.15.157:0/3275343202 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1fd8001cb0 con 0x7f1fd4038de0 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe0ebf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe0ebf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe0ebf640 10 monclient: dump: 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:00.360 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe4125640 10 monclient: shutdown 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe4125640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.379+0000 7f1fe4125640 1 -- 172.21.15.157:0/3275343202 >> v1:172.21.15.195:6789/0 conn(0x7f1fd4038de0 legacy=0x7f1fd40cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.380+0000 7f1fe4125640 1 -- 172.21.15.157:0/3275343202 shutdown_connections 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.380+0000 7f1fe4125640 1 -- 172.21.15.157:0/3275343202 >> 172.21.15.157:0/3275343202 conn(0x7f1fd4005c20 msgr2=0x7f1fd4006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.380+0000 7f1fe4125640 1 -- 172.21.15.157:0/3275343202 shutdown_connections 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.380+0000 7f1fe4125640 1 -- 172.21.15.157:0/3275343202 wait complete. 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.381+0000 7f1fe4125640 10 monclient: build_initial_monmap 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.381+0000 7f1fe4125640 10 monclient: monmap: 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:17:00.361 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.381+0000 7f1fe4125640 1 librados: starting msgr at 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.381+0000 7f1fe4125640 1 librados: starting objecter 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.381+0000 7f1fe4125640 1 Processor -- start 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.382+0000 7f1fe4125640 1 -- start start 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.382+0000 7f1fe4125640 1 librados: setting wanted keys 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.382+0000 7f1fe4125640 1 librados: calling monclient init 2024-03-29T11:17:00.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.382+0000 7f1fe4125640 10 monclient: init 2024-03-29T11:17:00.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.382+0000 7f1fe4125640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:00.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.382+0000 7f1fe4125640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:00.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.382+0000 7f1fe4125640 10 monclient(hunting): picked mon.noname-a con 0x7f1fd4030680 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.382+0000 7f1fe4125640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd4038a00 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fe4125640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd4009790 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fe4125640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd400a1e0 con 0x7f1fd4030680 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fe4125640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd400a430 con 0x7f1fd4038a00 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fe4125640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd400a710 con 0x7f1fd4009790 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fe4125640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fe4125640 10 monclient(hunting): authenticate will time out at 1232.580048s 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fe26c2640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f1fd4009790 0x7f1fd416c7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:54514/0 (socket says 172.21.15.157:54514) 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fe26c2640 1 -- 172.21.15.157:0/1820050823 learned_addr learned my addr 172.21.15.157:0/1820050823 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fd27fc640 10 client.?.objecter ms_handle_connect 0x7f1fd4009790 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fd27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fd27fc640 10 client.?.objecter ms_handle_connect 0x7f1fd4038a00 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.383+0000 7f1fd27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707668924 0 0) 0x7f1fd400a710 con 0x7f1fd4009790 2024-03-29T11:17:00.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc0004eb0 con 0x7f1fd4009790 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968170748 0 0) 0x7f1fd400a430 con 0x7f1fd4038a00 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd400a710 con 0x7f1fd4038a00 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.384+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1648901351 0 0) 0x7f1fd400a710 con 0x7f1fd4038a00 2024-03-29T11:17:00.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fd400a430 con 0x7f1fd4038a00 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1fd8001610 con 0x7f1fd4038a00 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 10 monclient(hunting): dump: 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:00.369 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:00.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:00.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1931659482 0 0) 0x7f1fc0004eb0 con 0x7f1fd4009790 2024-03-29T11:17:00.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fd400a710 con 0x7f1fd4009790 2024-03-29T11:17:00.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1fdc04fdd0 con 0x7f1fd4009790 2024-03-29T11:17:00.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:00.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:00.370 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 10 monclient(hunting): dump: 2024-03-29T11:17:00.370 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:00.370 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:00.370 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1332381482 0 0) 0x7f1fd400a430 con 0x7f1fd4038a00 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 >> v1:172.21.15.157:6790/0 conn(0x7f1fd4009790 legacy=0x7f1fd416c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.385+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 >> v1:172.21.15.157:6789/0 conn(0x7f1fd4030680 legacy=0x7f1fd4164cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fd27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fd27fc640 1 monclient: found mon.b 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:00.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fd27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd4164790 con 0x7f1fd4038a00 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fd27fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fd27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fe4125640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fe4125640 10 monclient: _renew_subs 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fe4125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fe4125640 1 -- 172.21.15.157:0/1820050823 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd4164460 con 0x7f1fd4038a00 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fe4125640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fe4125640 10 monclient: _renew_subs 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fe4125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fe4125640 1 -- 172.21.15.157:0/1820050823 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd40c3330 con 0x7f1fd4038a00 2024-03-29T11:17:00.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.386+0000 7f1fe4125640 1 librados: init done 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.387+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd8004a90 con 0x7f1fd4038a00 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.387+0000 7f1fd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.387+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1fd8004e20 con 0x7f1fd4038a00 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.387+0000 7f1fd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.387+0000 7f1fd27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.387+0000 7f1fd3fff640 4 set_mon_vals no callback set 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.387+0000 7f1fd27fc640 10 monclient: dump: 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:00.373 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:00.374 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:00.374 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:00.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.387+0000 7f1fe4125640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:00.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.388+0000 7f1fe4125640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:00.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.388+0000 7f1fe4125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:00.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.388+0000 7f1fe4125640 1 -- 172.21.15.157:0/1820050823 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc4005080 con 0x7f1fd4038a00 2024-03-29T11:17:00.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.389+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f1fd8004f80 con 0x7f1fd4038a00 2024-03-29T11:17:00.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.390+0000 7f1fe1ec1640 10 monclient: get_auth_request con 0x7f1fc006dd30 auth_method 0 2024-03-29T11:17:00.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.390+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f1fd8054a20 con 0x7f1fd4038a00 2024-03-29T11:17:00.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.390+0000 7f1fd27fc640 10 client.4417.objecter ms_dispatch 0x7f1fdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:00.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.390+0000 7f1fd27fc640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:00.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.390+0000 7f1fd27fc640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:00.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.390+0000 7f1fd27fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-29T11:17:00.375 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.390+0000 7f1fd27fc640 10 client.4417.objecter ms_handle_connect 0x7f1fc006dd30 2024-03-29T11:17:00.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.399+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fd8050cc0 con 0x7f1fd4038a00 2024-03-29T11:17:00.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.399+0000 7f1fd27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:00.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.399+0000 7f1fd27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:00.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.789+0000 7f1fe4125640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:00.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.789+0000 7f1fe4125640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:00.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.789+0000 7f1fe4125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:00.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.789+0000 7f1fe4125640 1 -- 172.21.15.157:0/1820050823 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fc4005330 con 0x7f1fd4038a00 2024-03-29T11:17:00.768 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:17:00.769 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:00.769 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.791+0000 7f1fd27fc640 1 -- 172.21.15.157:0/1820050823 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f1fd8050e60 con 0x7f1fd4038a00 2024-03-29T11:17:00.769 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.791+0000 7f1fd27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:00.769 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.791+0000 7f1fd27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:00.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.796+0000 7f1fe4125640 10 librados: watch_flush enter 2024-03-29T11:17:00.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.796+0000 7f1fe4125640 10 librados: watch_flush exit 2024-03-29T11:17:00.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.796+0000 7f1fe4125640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-29T11:17:00.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.796+0000 7f1fe4125640 10 client.4417.objecter successfully canceled tick 2024-03-29T11:17:00.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.796+0000 7f1fe4125640 1 -- 172.21.15.157:0/1820050823 >> v1:172.21.15.157:6816/1747058247 conn(0x7f1fc006dd30 legacy=0x7f1fc00701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:00.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.797+0000 7f1fe4125640 10 monclient: shutdown 2024-03-29T11:17:00.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.797+0000 7f1fe4125640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:00.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.797+0000 7f1fe4125640 1 -- 172.21.15.157:0/1820050823 >> v1:172.21.15.195:6789/0 conn(0x7f1fd4038a00 legacy=0x7f1fd40c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:00.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.798+0000 7f1fe4125640 1 -- 172.21.15.157:0/1820050823 shutdown_connections 2024-03-29T11:17:00.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.798+0000 7f1fe4125640 1 -- 172.21.15.157:0/1820050823 >> 172.21.15.157:0/1820050823 conn(0x7f1fd4005c20 msgr2=0x7f1fd40c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:00.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.798+0000 7f1fe4125640 1 -- 172.21.15.157:0/1820050823 shutdown_connections 2024-03-29T11:17:00.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.798+0000 7f1fe4125640 1 -- 172.21.15.157:0/1820050823 wait complete. 2024-03-29T11:17:00.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:00.799+0000 7f1fe4125640 1 librados: shutdown 2024-03-29T11:17:00.795 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:07.602 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:07.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.789+0000 7f87766a8640 10 monclient: get_monmap_and_config 2024-03-29T11:17:07.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.789+0000 7f87766a8640 10 monclient: build_initial_monmap 2024-03-29T11:17:07.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.789+0000 7f87766a8640 10 monclient: monmap: 2024-03-29T11:17:07.767 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:07.767 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:07.767 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:17:07.790640+0000 2024-03-29T11:17:07.767 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:17:07.790640+0000 2024-03-29T11:17:07.767 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:07.767 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:07.767 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:07.768 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:17:07.768 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:17:07.768 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.790+0000 7f87766a8640 1 Processor -- start 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.790+0000 7f87766a8640 1 -- start start 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.790+0000 7f87766a8640 10 monclient: init 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.791+0000 7f87766a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.791+0000 7f87766a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.791+0000 7f87766a8640 10 monclient(hunting): picked mon.noname-c con 0x7f87700b0500 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.791+0000 7f87766a8640 10 monclient(hunting): picked mon.noname-b con 0x7f87700b08e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.791+0000 7f87766a8640 10 monclient(hunting): picked mon.noname-a con 0x7f87700aeca0 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.791+0000 7f87766a8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8770140620 con 0x7f87700aeca0 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.791+0000 7f87766a8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877013daf0 con 0x7f87700b0500 2024-03-29T11:17:07.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.791+0000 7f87766a8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87700ae180 con 0x7f87700b08e0 2024-03-29T11:17:07.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.792+0000 7f87766a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:07.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.792+0000 7f87766a8640 10 monclient(hunting): authenticate will time out at 1239.988891s 2024-03-29T11:17:07.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.791+0000 7f876ffff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f87700b0500 0x7f877013ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:59518/0 (socket says 172.21.15.157:59518) 2024-03-29T11:17:07.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.792+0000 7f876ffff640 1 -- 172.21.15.157:0/3391878427 learned_addr learned my addr 172.21.15.157:0/3391878427 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:07.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.792+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028414366 0 0) 0x7f877013daf0 con 0x7f87700b0500 2024-03-29T11:17:07.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.792+0000 7f876effd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:07.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.792+0000 7f876effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:07.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.792+0000 7f876effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:07.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.792+0000 7f876effd640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:17:07.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.792+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8758003520 con 0x7f87700b0500 2024-03-29T11:17:07.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754639899 0 0) 0x7f8770140620 con 0x7f87700aeca0 2024-03-29T11:17:07.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:07.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:07.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:07.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:17:07.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f877013daf0 con 0x7f87700aeca0 2024-03-29T11:17:07.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1679787077 0 0) 0x7f8758003520 con 0x7f87700b0500 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 >> v1:172.21.15.157:6790/0 conn(0x7f87700b08e0 legacy=0x7f877013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 >> v1:172.21.15.157:6789/0 conn(0x7f87700aeca0 legacy=0x7f877014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 1 monclient: found mon.noname-c 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87700accf0 con 0x7f87700b0500 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient: _finish_auth 0 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f8760001690 con 0x7f87700b0500 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:07.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f876effd640 10 monclient: dump: 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f87766a8640 5 monclient: authenticate success, global_id 4420 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.793+0000 7f87766a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.794+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87600018b0 con 0x7f87700b0500 2024-03-29T11:17:07.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.794+0000 7f876effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.794+0000 7f876effd640 1 -- 172.21.15.157:0/3391878427 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f8760001c40 con 0x7f87700b0500 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.794+0000 7f87766a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.794+0000 7f87766a8640 4 set_mon_vals no callback set 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.794+0000 7f876effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.795+0000 7f876effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.795+0000 7f876effd640 10 monclient: dump: 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:07.777 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:07.778 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:07.778 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:07.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.795+0000 7f87766a8640 10 monclient: shutdown 2024-03-29T11:17:07.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.795+0000 7f87766a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:07.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.795+0000 7f87766a8640 1 -- 172.21.15.157:0/3391878427 >> v1:172.21.15.195:6789/0 conn(0x7f87700b0500 legacy=0x7f877013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:07.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.796+0000 7f87766a8640 1 -- 172.21.15.157:0/3391878427 shutdown_connections 2024-03-29T11:17:07.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.796+0000 7f87766a8640 1 -- 172.21.15.157:0/3391878427 >> 172.21.15.157:0/3391878427 conn(0x7f877006b660 msgr2=0x7f877006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:07.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.796+0000 7f87766a8640 1 -- 172.21.15.157:0/3391878427 shutdown_connections 2024-03-29T11:17:07.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.796+0000 7f87766a8640 1 -- 172.21.15.157:0/3391878427 wait complete. 2024-03-29T11:17:07.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.797+0000 7f87766a8640 10 monclient: build_initial_monmap 2024-03-29T11:17:07.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.797+0000 7f87766a8640 10 monclient: monmap: 2024-03-29T11:17:07.782 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:07.782 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:07.782 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:17:07.782 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:17:07.782 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:07.782 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.798+0000 7f87766a8640 1 librados: starting msgr at 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.798+0000 7f87766a8640 1 librados: starting objecter 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.798+0000 7f87766a8640 1 Processor -- start 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.799+0000 7f87766a8640 1 -- start start 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.799+0000 7f87766a8640 1 librados: setting wanted keys 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.799+0000 7f87766a8640 1 librados: calling monclient init 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.799+0000 7f87766a8640 10 monclient: init 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f87766a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f87766a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:07.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f87766a8640 10 monclient(hunting): picked mon.noname-c con 0x7f87700aeca0 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f87766a8640 10 monclient(hunting): picked mon.noname-a con 0x7f87700b0500 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f87766a8640 10 monclient(hunting): picked mon.noname-b con 0x7f87701c81a0 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f87766a8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87701c9bc0 con 0x7f87700b0500 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f87766a8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87701c8dd0 con 0x7f87701c81a0 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f87766a8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877013c5b0 con 0x7f87700aeca0 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f87766a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f87766a8640 10 monclient(hunting): authenticate will time out at 1239.997582s 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f876f7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f87700b0500 0x7f877013b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:40252/0 (socket says 172.21.15.157:40252) 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.800+0000 7f876f7fe640 1 -- 172.21.15.157:0/2364901296 learned_addr learned my addr 172.21.15.157:0/2364901296 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 client.?.objecter ms_handle_connect 0x7f87700b0500 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 client.?.objecter ms_handle_connect 0x7f87700aeca0 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:07.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 client.?.objecter ms_handle_connect 0x7f87701c81a0 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305879678 0 0) 0x7f87701c9bc0 con 0x7f87700b0500 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f875c004eb0 con 0x7f87700b0500 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675875380 0 0) 0x7f87701c8dd0 con 0x7f87701c81a0 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:17:07.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.801+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87701c9bc0 con 0x7f87701c81a0 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308937089 0 0) 0x7f877013c5b0 con 0x7f87700aeca0 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87701c8dd0 con 0x7f87700aeca0 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2059697694 0 0) 0x7f875c004eb0 con 0x7f87700b0500 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f877013c5b0 con 0x7f87700b0500 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f8764001960 con 0x7f87700b0500 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): dump: 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:07.786 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2654706985 0 0) 0x7f87701c9bc0 con 0x7f87701c81a0 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f875c004eb0 con 0x7f87701c81a0 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f8768001c30 con 0x7f87701c81a0 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:07.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 10 monclient(hunting): dump: 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4691475 0 0) 0x7f875c004eb0 con 0x7f87701c81a0 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.802+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 >> v1:172.21.15.157:6790/0 conn(0x7f87700aeca0 legacy=0x7f87701c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 >> v1:172.21.15.157:6789/0 conn(0x7f87700b0500 legacy=0x7f877013b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:17:07.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 1 monclient: found mon.b 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87701c9df0 con 0x7f87701c81a0 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 10 monclient: _finish_auth 0 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f87766a8640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f87766a8640 10 monclient: _renew_subs 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f87766a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f87766a8640 1 -- 172.21.15.157:0/2364901296 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87701ca0d0 con 0x7f87701c81a0 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f87766a8640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f87766a8640 10 monclient: _renew_subs 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f87766a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:07.789 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f87766a8640 1 -- 172.21.15.157:0/2364901296 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8770070550 con 0x7f87701c81a0 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f87766a8640 1 librados: init done 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8768002040 con 0x7f87701c81a0 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f8768003b30 con 0x7f87701c81a0 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876cff9640 10 monclient: dump: 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:07.790 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:07.791 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:07.791 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:07.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.803+0000 7f876dffb640 4 set_mon_vals no callback set 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.805+0000 7f87766a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.805+0000 7f87766a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.805+0000 7f87766a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.805+0000 7f87766a8640 1 -- 172.21.15.157:0/2364901296 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8750005080 con 0x7f87701c81a0 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.806+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f8768003c90 con 0x7f87701c81a0 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.807+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f8768083a20 con 0x7f87701c81a0 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.807+0000 7f876cff9640 10 client.4423.objecter ms_dispatch 0x7f8770000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.807+0000 7f876cff9640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.807+0000 7f876cff9640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.807+0000 7f876ffff640 10 monclient: get_auth_request con 0x7f875c06dfe0 auth_method 0 2024-03-29T11:17:07.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.807+0000 7f876cff9640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:17:07.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.807+0000 7f876cff9640 10 client.4423.objecter ms_handle_connect 0x7f875c06dfe0 2024-03-29T11:17:07.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.813+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f876804f360 con 0x7f87701c81a0 2024-03-29T11:17:07.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.813+0000 7f876cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:07.793 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:07.813+0000 7f876cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:08.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.253+0000 7f87766a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:08.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.253+0000 7f87766a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:08.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.253+0000 7f87766a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:08.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.253+0000 7f87766a8640 1 -- 172.21.15.157:0/2364901296 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8750005330 con 0x7f87701c81a0 2024-03-29T11:17:08.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.254+0000 7f876cff9640 1 -- 172.21.15.157:0/2364901296 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f8768051290 con 0x7f87701c81a0 2024-03-29T11:17:08.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.254+0000 7f876cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:08.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.254+0000 7f876cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:08.232 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:17:08.232 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:08.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.257+0000 7f87766a8640 10 librados: watch_flush enter 2024-03-29T11:17:08.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.258+0000 7f87766a8640 10 librados: watch_flush exit 2024-03-29T11:17:08.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.258+0000 7f87766a8640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:17:08.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.258+0000 7f87766a8640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:17:08.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.258+0000 7f87766a8640 1 -- 172.21.15.157:0/2364901296 >> v1:172.21.15.157:6816/1747058247 conn(0x7f875c06dfe0 legacy=0x7f875c0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:08.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.258+0000 7f87766a8640 10 monclient: shutdown 2024-03-29T11:17:08.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.258+0000 7f87766a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:08.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.258+0000 7f87766a8640 1 -- 172.21.15.157:0/2364901296 >> v1:172.21.15.195:6789/0 conn(0x7f87701c81a0 legacy=0x7f877013bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:08.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.258+0000 7f87766a8640 1 -- 172.21.15.157:0/2364901296 shutdown_connections 2024-03-29T11:17:08.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.258+0000 7f87766a8640 1 -- 172.21.15.157:0/2364901296 >> 172.21.15.157:0/2364901296 conn(0x7f877006b660 msgr2=0x7f877006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:08.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.258+0000 7f87766a8640 1 -- 172.21.15.157:0/2364901296 shutdown_connections 2024-03-29T11:17:08.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.259+0000 7f87766a8640 1 -- 172.21.15.157:0/2364901296 wait complete. 2024-03-29T11:17:08.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:08.259+0000 7f87766a8640 1 librados: shutdown 2024-03-29T11:17:08.253 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:15.060 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:15.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.217+0000 7f1c61ba8640 10 monclient: get_monmap_and_config 2024-03-29T11:17:15.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.217+0000 7f1c61ba8640 10 monclient: build_initial_monmap 2024-03-29T11:17:15.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.217+0000 7f1c61ba8640 10 monclient: monmap: 2024-03-29T11:17:15.195 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:15.195 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:15.195 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:17:15.218389+0000 2024-03-29T11:17:15.195 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:17:15.218389+0000 2024-03-29T11:17:15.196 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:15.196 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:15.196 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:15.196 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:17:15.196 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:17:15.196 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:15.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.218+0000 7f1c61ba8640 1 Processor -- start 2024-03-29T11:17:15.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.218+0000 7f1c61ba8640 1 -- start start 2024-03-29T11:17:15.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.218+0000 7f1c61ba8640 10 monclient: init 2024-03-29T11:17:15.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c61ba8640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:15.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c61ba8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:15.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c61ba8640 10 monclient(hunting): picked mon.noname-b con 0x7f1c5c0b0770 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:15.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c61ba8640 10 monclient(hunting): picked mon.noname-a con 0x7f1c5c14a700 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:15.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c61ba8640 10 monclient(hunting): picked mon.noname-c con 0x7f1c5c0ac5a0 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:15.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c61ba8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c0aba70 con 0x7f1c5c14a700 2024-03-29T11:17:15.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c61ba8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c0aecb0 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c61ba8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c0af150 con 0x7f1c5c0b0770 2024-03-29T11:17:15.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c61ba8640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c61ba8640 10 monclient(hunting): authenticate will time out at 1247.416379s 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c5b7fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f1c5c0b0770 0x7f1c5c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:47496/0 (socket says 172.21.15.157:47496) 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c5b7fe640 1 -- 172.21.15.157:0/2629548636 learned_addr learned my addr 172.21.15.157:0/2629548636 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353498831 0 0) 0x7f1c5c0af150 con 0x7f1c5c0b0770 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c5a7fc640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.219+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c44003520 con 0x7f1c5c0b0770 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.220+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325748326 0 0) 0x7f1c5c0aecb0 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.220+0000 7f1c5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:15.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.220+0000 7f1c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:15.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.220+0000 7f1c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:15.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.220+0000 7f1c5a7fc640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:17:15.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.220+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c5c0af150 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.220+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1564895289 0 0) 0x7f1c44003520 con 0x7f1c5c0b0770 2024-03-29T11:17:15.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.220+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 >> v1:172.21.15.195:6789/0 conn(0x7f1c5c0ac5a0 legacy=0x7f1c5c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:15.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.220+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 >> v1:172.21.15.157:6789/0 conn(0x7f1c5c14a700 legacy=0x7f1c5c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:15.201 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:15.201 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:17:15.201 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:15.201 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:17:15.201 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c0ac9d0 con 0x7f1c5c0b0770 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1c50001690 con 0x7f1c5c0b0770 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: dump: 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:15.202 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c61ba8640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c61ba8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c500017f0 con 0x7f1c5c0b0770 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 1 -- 172.21.15.157:0/2629548636 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1c50001c00 con 0x7f1c5c0b0770 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c5a7fc640 10 monclient: dump: 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:15.203 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c61ba8640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.222+0000 7f1c61ba8640 4 set_mon_vals no callback set 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.223+0000 7f1c61ba8640 10 monclient: shutdown 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.223+0000 7f1c61ba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.223+0000 7f1c61ba8640 1 -- 172.21.15.157:0/2629548636 >> v1:172.21.15.157:6790/0 conn(0x7f1c5c0b0770 legacy=0x7f1c5c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.223+0000 7f1c61ba8640 1 -- 172.21.15.157:0/2629548636 shutdown_connections 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.223+0000 7f1c61ba8640 1 -- 172.21.15.157:0/2629548636 >> 172.21.15.157:0/2629548636 conn(0x7f1c5c06b660 msgr2=0x7f1c5c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.223+0000 7f1c61ba8640 1 -- 172.21.15.157:0/2629548636 shutdown_connections 2024-03-29T11:17:15.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.223+0000 7f1c61ba8640 1 -- 172.21.15.157:0/2629548636 wait complete. 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.224+0000 7f1c61ba8640 10 monclient: build_initial_monmap 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.224+0000 7f1c61ba8640 10 monclient: monmap: 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:17:15.205 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:15.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.224+0000 7f1c61ba8640 1 librados: starting msgr at 2024-03-29T11:17:15.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.224+0000 7f1c61ba8640 1 librados: starting objecter 2024-03-29T11:17:15.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.224+0000 7f1c61ba8640 1 Processor -- start 2024-03-29T11:17:15.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.225+0000 7f1c61ba8640 1 -- start start 2024-03-29T11:17:15.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.225+0000 7f1c61ba8640 1 librados: setting wanted keys 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.225+0000 7f1c61ba8640 1 librados: calling monclient init 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.225+0000 7f1c61ba8640 10 monclient: init 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.225+0000 7f1c61ba8640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.225+0000 7f1c61ba8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c61ba8640 10 monclient(hunting): picked mon.noname-c con 0x7f1c5c0ac5a0 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c61ba8640 10 monclient(hunting): picked mon.noname-b con 0x7f1c5c0b0770 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c61ba8640 10 monclient(hunting): picked mon.noname-a con 0x7f1c5c14a700 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c61ba8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c126510 con 0x7f1c5c14a700 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c61ba8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c1284d0 con 0x7f1c5c0b0770 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c61ba8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c1287b0 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c61ba8640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c61ba8640 10 monclient(hunting): authenticate will time out at 1247.422935s 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c5b7fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f1c5c0ac5a0 0x7f1c5c145150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:47714/0 (socket says 172.21.15.157:47714) 2024-03-29T11:17:15.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c5b7fe640 1 -- 172.21.15.157:0/44306038 learned_addr learned my addr 172.21.15.157:0/44306038 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c43fff640 10 client.?.objecter ms_handle_connect 0x7f1c5c0ac5a0 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c43fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840558030 0 0) 0x7f1c5c1287b0 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c43fff640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.226+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c48004eb0 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.227+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4093242514 0 0) 0x7f1c48004eb0 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.227+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c48005300 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.227+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1c50001c00 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.227+0000 7f1c43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.227+0000 7f1c43fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:15.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.227+0000 7f1c43fff640 10 monclient(hunting): dump: 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.227+0000 7f1c43fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.227+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2595328373 0 0) 0x7f1c48005300 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.227+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 >> v1:172.21.15.195:6789/0 conn(0x7f1c5c0b0770 legacy=0x7f1c5c1269a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.227+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 >> v1:172.21.15.157:6789/0 conn(0x7f1c5c14a700 legacy=0x7f1c5c1270b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:15.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.228+0000 7f1c43fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.228+0000 7f1c43fff640 1 monclient: found mon.c 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.228+0000 7f1c43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.228+0000 7f1c43fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.228+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c1260e0 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.228+0000 7f1c43fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.228+0000 7f1c43fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.228+0000 7f1c43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c61ba8640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c61ba8640 10 monclient: _renew_subs 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c61ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c61ba8640 1 -- 172.21.15.157:0/44306038 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5c1df960 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c61ba8640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c61ba8640 10 monclient: _renew_subs 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c61ba8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:15.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c61ba8640 1 -- 172.21.15.157:0/44306038 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5c1df120 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c61ba8640 1 librados: init done 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c50004b10 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1c50001c00 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c43fff640 10 monclient: dump: 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:15.214 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:15.215 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:15.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.229+0000 7f1c59ffb640 4 set_mon_vals no callback set 2024-03-29T11:17:15.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.230+0000 7f1c41ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:15.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.230+0000 7f1c41ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:15.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.230+0000 7f1c41ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:15.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.230+0000 7f1c41ffb640 1 -- 172.21.15.157:0/44306038 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c34005080 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.232+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f1c50004d80 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.232+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f1c50084950 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.232+0000 7f1c43fff640 10 client.4454.objecter ms_dispatch 0x7f1c5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:15.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.232+0000 7f1c43fff640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:15.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.232+0000 7f1c43fff640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:15.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.233+0000 7f1c43fff640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T11:17:15.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.233+0000 7f1c5affd640 10 monclient: get_auth_request con 0x7f1c4806e040 auth_method 0 2024-03-29T11:17:15.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.233+0000 7f1c43fff640 10 client.4454.objecter ms_handle_connect 0x7f1c4806e040 2024-03-29T11:17:15.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.238+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c50050230 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.238+0000 7f1c43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:15.216 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.238+0000 7f1c43fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:15.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.540+0000 7f1c41ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:15.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.541+0000 7f1c41ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:15.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.541+0000 7f1c41ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:15.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.541+0000 7f1c41ffb640 1 -- 172.21.15.157:0/44306038 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c34005330 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.542+0000 7f1c43fff640 1 -- 172.21.15.157:0/44306038 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f1c50052160 con 0x7f1c5c0ac5a0 2024-03-29T11:17:15.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.542+0000 7f1c43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:15.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.542+0000 7f1c43fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:15.520 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:17:15.520 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:15.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.545+0000 7f1c41ffb640 10 librados: watch_flush enter 2024-03-29T11:17:15.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.545+0000 7f1c41ffb640 10 librados: watch_flush exit 2024-03-29T11:17:15.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.545+0000 7f1c41ffb640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T11:17:15.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.545+0000 7f1c41ffb640 10 client.4454.objecter successfully canceled tick 2024-03-29T11:17:15.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.545+0000 7f1c41ffb640 1 -- 172.21.15.157:0/44306038 >> v1:172.21.15.157:6816/1747058247 conn(0x7f1c4806e040 legacy=0x7f1c48070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:15.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.545+0000 7f1c41ffb640 10 monclient: shutdown 2024-03-29T11:17:15.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.545+0000 7f1c41ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:15.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.545+0000 7f1c41ffb640 1 -- 172.21.15.157:0/44306038 >> v1:172.21.15.157:6790/0 conn(0x7f1c5c0ac5a0 legacy=0x7f1c5c145150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:15.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.545+0000 7f1c41ffb640 1 -- 172.21.15.157:0/44306038 shutdown_connections 2024-03-29T11:17:15.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.545+0000 7f1c41ffb640 1 -- 172.21.15.157:0/44306038 >> 172.21.15.157:0/44306038 conn(0x7f1c5c06b660 msgr2=0x7f1c5c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:15.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.546+0000 7f1c41ffb640 1 -- 172.21.15.157:0/44306038 shutdown_connections 2024-03-29T11:17:15.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.546+0000 7f1c41ffb640 1 -- 172.21.15.157:0/44306038 wait complete. 2024-03-29T11:17:15.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:15.546+0000 7f1c41ffb640 1 librados: shutdown 2024-03-29T11:17:15.538 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:22.345 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:22.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.521+0000 7f4f97904640 10 monclient: get_monmap_and_config 2024-03-29T11:17:22.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.521+0000 7f4f97904640 10 monclient: build_initial_monmap 2024-03-29T11:17:22.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.521+0000 7f4f97904640 10 monclient: monmap: 2024-03-29T11:17:22.498 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:22.499 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:22.499 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:17:22.522036+0000 2024-03-29T11:17:22.499 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:17:22.522036+0000 2024-03-29T11:17:22.499 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:22.499 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:22.499 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:22.499 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:17:22.499 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:17:22.499 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:22.499 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.522+0000 7f4f97904640 1 Processor -- start 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 1 -- start start 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 10 monclient: init 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 10 monclient(hunting): picked mon.noname-a con 0x7f4f901457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9013f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 10 monclient(hunting): picked mon.noname-b con 0x7f4f900a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f900aaef0 con 0x7f4f901457f0 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f90145bd0 con 0x7f4f9013f650 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f90145eb0 con 0x7f4f900a9d70 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:22.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.523+0000 7f4f97904640 10 monclient(hunting): authenticate will time out at 1254.719810s 2024-03-29T11:17:22.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.524+0000 7f4f95ea1640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f4f900a9d70 0x7f4f9014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:54358/0 (socket says 172.21.15.157:54358) 2024-03-29T11:17:22.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.524+0000 7f4f95ea1640 1 -- 172.21.15.157:0/2925396262 learned_addr learned my addr 172.21.15.157:0/2925396262 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:22.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.524+0000 7f4f87fff640 1 -- 172.21.15.157:0/2925396262 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269976735 0 0) 0x7f4f90145eb0 con 0x7f4f900a9d70 2024-03-29T11:17:22.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.524+0000 7f4f87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:22.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.524+0000 7f4f87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:22.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.524+0000 7f4f87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:22.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.524+0000 7f4f87fff640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:17:22.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.524+0000 7f4f87fff640 1 -- 172.21.15.157:0/2925396262 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f7c003520 con 0x7f4f900a9d70 2024-03-29T11:17:22.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.524+0000 7f4f87fff640 1 -- 172.21.15.157:0/2925396262 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 866814489 0 0) 0x7f4f7c003520 con 0x7f4f900a9d70 2024-03-29T11:17:22.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.525+0000 7f4f87fff640 1 -- 172.21.15.157:0/2925396262 >> v1:172.21.15.195:6789/0 conn(0x7f4f9013f650 legacy=0x7f4f900b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:22.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 1 -- 172.21.15.157:0/2925396262 >> v1:172.21.15.157:6789/0 conn(0x7f4f901457f0 legacy=0x7f4f9013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:22.507 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 1 monclient: found mon.noname-b 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 1 -- 172.21.15.157:0/2925396262 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f90146340 con 0x7f4f900a9d70 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 1 -- 172.21.15.157:0/2925396262 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4f88001a10 con 0x7f4f900a9d70 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f87fff640 10 monclient: dump: 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:22.508 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:22.509 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:22.509 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:22.509 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:22.509 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:22.509 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:22.509 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:22.509 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:22.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f97904640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:17:22.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.531+0000 7f4f97904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.533+0000 7f4f87fff640 1 -- 172.21.15.157:0/2925396262 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f88001ca0 con 0x7f4f900a9d70 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.533+0000 7f4f87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.533+0000 7f4f87fff640 1 -- 172.21.15.157:0/2925396262 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4f88002030 con 0x7f4f900a9d70 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.533+0000 7f4f87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.533+0000 7f4f87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.533+0000 7f4f87fff640 10 monclient: dump: 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:22.510 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:22.511 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:22.511 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:22.511 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:22.511 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:22.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.534+0000 7f4f97904640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:22.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.534+0000 7f4f97904640 4 set_mon_vals no callback set 2024-03-29T11:17:22.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.534+0000 7f4f97904640 10 monclient: shutdown 2024-03-29T11:17:22.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.534+0000 7f4f97904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:22.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.535+0000 7f4f97904640 1 -- 172.21.15.157:0/2925396262 >> v1:172.21.15.157:6790/0 conn(0x7f4f900a9d70 legacy=0x7f4f9014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:22.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.535+0000 7f4f97904640 1 -- 172.21.15.157:0/2925396262 shutdown_connections 2024-03-29T11:17:22.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.535+0000 7f4f97904640 1 -- 172.21.15.157:0/2925396262 >> 172.21.15.157:0/2925396262 conn(0x7f4f90066a00 msgr2=0x7f4f90066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:22.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.535+0000 7f4f97904640 1 -- 172.21.15.157:0/2925396262 shutdown_connections 2024-03-29T11:17:22.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.535+0000 7f4f97904640 1 -- 172.21.15.157:0/2925396262 wait complete. 2024-03-29T11:17:22.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.536+0000 7f4f97904640 10 monclient: build_initial_monmap 2024-03-29T11:17:22.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.536+0000 7f4f97904640 10 monclient: monmap: 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.537+0000 7f4f97904640 1 librados: starting msgr at 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.537+0000 7f4f97904640 1 librados: starting objecter 2024-03-29T11:17:22.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.537+0000 7f4f97904640 1 Processor -- start 2024-03-29T11:17:22.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.537+0000 7f4f97904640 1 -- start start 2024-03-29T11:17:22.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.537+0000 7f4f97904640 1 librados: setting wanted keys 2024-03-29T11:17:22.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.537+0000 7f4f97904640 1 librados: calling monclient init 2024-03-29T11:17:22.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.537+0000 7f4f97904640 10 monclient: init 2024-03-29T11:17:22.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.538+0000 7f4f97904640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:22.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.538+0000 7f4f97904640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:22.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.538+0000 7f4f97904640 10 monclient(hunting): picked mon.noname-a con 0x7f4f900a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:22.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.538+0000 7f4f97904640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9013f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:22.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.539+0000 7f4f94e9f640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f4f9013f650 0x7f4f901425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:54894/0 (socket says 172.21.15.157:54894) 2024-03-29T11:17:22.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.539+0000 7f4f97904640 10 monclient(hunting): picked mon.noname-b con 0x7f4f901457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:22.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.539+0000 7f4f97904640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f90141930 con 0x7f4f900a9d70 2024-03-29T11:17:22.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.539+0000 7f4f97904640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f90140280 con 0x7f4f901457f0 2024-03-29T11:17:22.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.539+0000 7f4f97904640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f90141b60 con 0x7f4f9013f650 2024-03-29T11:17:22.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.539+0000 7f4f97904640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:22.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.539+0000 7f4f97904640 10 monclient(hunting): authenticate will time out at 1254.735752s 2024-03-29T11:17:22.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.539+0000 7f4f94e9f640 1 -- 172.21.15.157:0/1257233989 learned_addr learned my addr 172.21.15.157:0/1257233989 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:22.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.541+0000 7f4f85ffb640 10 client.?.objecter ms_handle_connect 0x7f4f9013f650 2024-03-29T11:17:22.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.541+0000 7f4f85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:22.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.542+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922894445 0 0) 0x7f4f90141b60 con 0x7f4f9013f650 2024-03-29T11:17:22.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.542+0000 7f4f85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:22.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.542+0000 7f4f85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:22.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.542+0000 7f4f85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:22.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.542+0000 7f4f85ffb640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:17:22.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.542+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f74004eb0 con 0x7f4f9013f650 2024-03-29T11:17:22.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.542+0000 7f4f85ffb640 10 client.?.objecter ms_handle_connect 0x7f4f901457f0 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.542+0000 7f4f85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115724109 0 0) 0x7f4f90140280 con 0x7f4f901457f0 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f740051d0 con 0x7f4f901457f0 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070784504 0 0) 0x7f4f74004eb0 con 0x7f4f9013f650 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f90140280 con 0x7f4f9013f650 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4f80001830 con 0x7f4f9013f650 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:22.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763236665 0 0) 0x7f4f740051d0 con 0x7f4f901457f0 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f74004eb0 con 0x7f4f901457f0 2024-03-29T11:17:22.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4f880018d0 con 0x7f4f901457f0 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.543+0000 7f4f85ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 454284701 0 0) 0x7f4f74004eb0 con 0x7f4f901457f0 2024-03-29T11:17:22.524 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 >> v1:172.21.15.157:6790/0 conn(0x7f4f9013f650 legacy=0x7f4f901425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 >> v1:172.21.15.157:6789/0 conn(0x7f4f900a9d70 legacy=0x7f4f90138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 1 monclient: found mon.b 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f90141e40 con 0x7f4f901457f0 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f97904640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f97904640 10 monclient: _renew_subs 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f97904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f97904640 1 -- 172.21.15.157:0/1257233989 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f90140500 con 0x7f4f901457f0 2024-03-29T11:17:22.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f88001a30 con 0x7f4f901457f0 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4f88005110 con 0x7f4f901457f0 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f85ffb640 10 monclient: dump: 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:22.526 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:22.527 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:22.527 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.544+0000 7f4f877fe640 4 set_mon_vals no callback set 2024-03-29T11:17:22.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.545+0000 7f4f97904640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:22.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.545+0000 7f4f97904640 10 monclient: _renew_subs 2024-03-29T11:17:22.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.545+0000 7f4f97904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:22.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.545+0000 7f4f97904640 1 -- 172.21.15.157:0/1257233989 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f90142ce0 con 0x7f4f901457f0 2024-03-29T11:17:22.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.545+0000 7f4f97904640 1 librados: init done 2024-03-29T11:17:22.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.546+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f4f8801b0e0 con 0x7f4f901457f0 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.547+0000 7f4f956a0640 10 monclient: get_auth_request con 0x7f4f740767a0 auth_method 0 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.547+0000 7f4f85ffb640 10 client.4429.objecter ms_handle_connect 0x7f4f740767a0 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.550+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f4f88054c50 con 0x7f4f901457f0 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.550+0000 7f4f85ffb640 10 client.4429.objecter ms_dispatch 0x7f4f90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.550+0000 7f4f85ffb640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.550+0000 7f4f85ffb640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.550+0000 7f4f85ffb640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.551+0000 7f4f97904640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.551+0000 7f4f97904640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.551+0000 7f4f97904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:22.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.551+0000 7f4f97904640 1 -- 172.21.15.157:0/1257233989 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f90075f10 con 0x7f4f901457f0 2024-03-29T11:17:22.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.559+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f88050760 con 0x7f4f901457f0 2024-03-29T11:17:22.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.559+0000 7f4f85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:22.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.559+0000 7f4f85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:22.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.879+0000 7f4f97904640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:22.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.879+0000 7f4f97904640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:22.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.879+0000 7f4f97904640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:22.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.879+0000 7f4f97904640 1 -- 172.21.15.157:0/1257233989 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f900b3680 con 0x7f4f901457f0 2024-03-29T11:17:22.858 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.882+0000 7f4f85ffb640 1 -- 172.21.15.157:0/1257233989 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f4f88052690 con 0x7f4f901457f0 2024-03-29T11:17:22.858 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.882+0000 7f4f85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:22.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.882+0000 7f4f85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:22.859 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:17:22.859 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:22.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.886+0000 7f4f97904640 10 librados: watch_flush enter 2024-03-29T11:17:22.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.886+0000 7f4f97904640 10 librados: watch_flush exit 2024-03-29T11:17:22.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.886+0000 7f4f97904640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-29T11:17:22.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.886+0000 7f4f97904640 10 client.4429.objecter successfully canceled tick 2024-03-29T11:17:22.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.886+0000 7f4f97904640 1 -- 172.21.15.157:0/1257233989 >> v1:172.21.15.157:6816/1747058247 conn(0x7f4f740767a0 legacy=0x7f4f74078c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:22.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.886+0000 7f4f97904640 10 monclient: shutdown 2024-03-29T11:17:22.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.886+0000 7f4f97904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:22.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.886+0000 7f4f97904640 1 -- 172.21.15.157:0/1257233989 >> v1:172.21.15.195:6789/0 conn(0x7f4f901457f0 legacy=0x7f4f901e1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:22.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.887+0000 7f4f97904640 1 -- 172.21.15.157:0/1257233989 shutdown_connections 2024-03-29T11:17:22.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.887+0000 7f4f97904640 1 -- 172.21.15.157:0/1257233989 >> 172.21.15.157:0/1257233989 conn(0x7f4f90066a00 msgr2=0x7f4f900b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:22.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.887+0000 7f4f97904640 1 -- 172.21.15.157:0/1257233989 shutdown_connections 2024-03-29T11:17:22.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.887+0000 7f4f97904640 1 -- 172.21.15.157:0/1257233989 wait complete. 2024-03-29T11:17:22.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:22.887+0000 7f4f97904640 1 librados: shutdown 2024-03-29T11:17:22.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:29.688 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:29.838 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.860+0000 7f5586977640 10 monclient: get_monmap_and_config 2024-03-29T11:17:29.838 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.860+0000 7f5586977640 10 monclient: build_initial_monmap 2024-03-29T11:17:29.838 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.860+0000 7f5586977640 10 monclient: monmap: 2024-03-29T11:17:29.838 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:29.838 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:29.838 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:17:29.861854+0000 2024-03-29T11:17:29.838 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:17:29.861854+0000 2024-03-29T11:17:29.838 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:29.839 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:29.839 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:29.839 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:17:29.839 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:17:29.839 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:29.840 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.862+0000 7f5586977640 1 Processor -- start 2024-03-29T11:17:29.840 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.862+0000 7f5586977640 1 -- start start 2024-03-29T11:17:29.840 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.862+0000 7f5586977640 10 monclient: init 2024-03-29T11:17:29.840 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.863+0000 7f5586977640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:29.840 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.863+0000 7f5586977640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:29.841 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.863+0000 7f5586977640 10 monclient(hunting): picked mon.noname-a con 0x7f55801457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:29.841 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.863+0000 7f5586977640 10 monclient(hunting): picked mon.noname-c con 0x7f558013f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:29.841 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.863+0000 7f5586977640 10 monclient(hunting): picked mon.noname-b con 0x7f55800a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:29.841 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.863+0000 7f5586977640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55800aaef0 con 0x7f55801457f0 2024-03-29T11:17:29.841 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.863+0000 7f5586977640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5580145bd0 con 0x7f558013f650 2024-03-29T11:17:29.841 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.863+0000 7f5586977640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5580145eb0 con 0x7f55800a9d70 2024-03-29T11:17:29.841 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.863+0000 7f5586977640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:29.841 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.863+0000 7f5586977640 10 monclient(hunting): authenticate will time out at 1262.060284s 2024-03-29T11:17:29.841 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.864+0000 7f5584f14640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f55800a9d70 0x7f558014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:35472/0 (socket says 172.21.15.157:35472) 2024-03-29T11:17:29.841 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.864+0000 7f5584f14640 1 -- 172.21.15.157:0/1777425592 learned_addr learned my addr 172.21.15.157:0/1777425592 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:29.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.866+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790453591 0 0) 0x7f5580145eb0 con 0x7f55800a9d70 2024-03-29T11:17:29.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.866+0000 7f557effd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:29.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.866+0000 7f557effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:29.844 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.866+0000 7f557effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:29.845 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.866+0000 7f557effd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:17:29.845 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.866+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5568003520 con 0x7f55800a9d70 2024-03-29T11:17:29.845 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303500420 0 0) 0x7f5580145bd0 con 0x7f558013f650 2024-03-29T11:17:29.845 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:29.845 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:29.845 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:29.845 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:17:29.845 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5568003840 con 0x7f558013f650 2024-03-29T11:17:29.846 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2783778312 0 0) 0x7f5568003840 con 0x7f558013f650 2024-03-29T11:17:29.846 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 >> v1:172.21.15.157:6790/0 conn(0x7f55800a9d70 legacy=0x7f558014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.846 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 >> v1:172.21.15.157:6789/0 conn(0x7f55801457f0 legacy=0x7f558013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:29.846 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:29.846 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 1 monclient: found mon.noname-c 2024-03-29T11:17:29.846 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:29.846 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:17:29.846 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5580146340 con 0x7f558013f650 2024-03-29T11:17:29.846 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient: _finish_auth 0 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5574001630 con 0x7f558013f650 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f557effd640 10 monclient: dump: 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:29.847 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f5586977640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.867+0000 7f5586977640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.868+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5574001880 con 0x7f558013f650 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.868+0000 7f557effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.868+0000 7f5586977640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.868+0000 7f5586977640 4 set_mon_vals no callback set 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.868+0000 7f557effd640 1 -- 172.21.15.157:0/1777425592 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5574001c10 con 0x7f558013f650 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.868+0000 7f557effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.868+0000 7f557effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.868+0000 7f557effd640 10 monclient: dump: 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:29.848 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:29.849 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:29.849 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:29.849 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:29.849 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:29.849 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:29.849 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:29.850 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.869+0000 7f5586977640 10 monclient: shutdown 2024-03-29T11:17:29.850 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.869+0000 7f5586977640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.869+0000 7f5586977640 1 -- 172.21.15.157:0/1777425592 >> v1:172.21.15.195:6789/0 conn(0x7f558013f650 legacy=0x7f55800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.869+0000 7f5586977640 1 -- 172.21.15.157:0/1777425592 shutdown_connections 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.870+0000 7f5586977640 1 -- 172.21.15.157:0/1777425592 >> 172.21.15.157:0/1777425592 conn(0x7f5580066a00 msgr2=0x7f5580066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.870+0000 7f5586977640 1 -- 172.21.15.157:0/1777425592 shutdown_connections 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.870+0000 7f5586977640 1 -- 172.21.15.157:0/1777425592 wait complete. 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.871+0000 7f5586977640 10 monclient: build_initial_monmap 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.872+0000 7f5586977640 10 monclient: monmap: 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:29.851 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:29.852 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:17:29.852 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:17:29.852 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:29.852 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.872+0000 7f5586977640 1 librados: starting msgr at 2024-03-29T11:17:29.852 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.872+0000 7f5586977640 1 librados: starting objecter 2024-03-29T11:17:29.852 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.872+0000 7f5586977640 1 Processor -- start 2024-03-29T11:17:29.852 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.873+0000 7f5586977640 1 -- start start 2024-03-29T11:17:29.852 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.873+0000 7f5586977640 1 librados: setting wanted keys 2024-03-29T11:17:29.852 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.873+0000 7f5586977640 1 librados: calling monclient init 2024-03-29T11:17:29.852 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.873+0000 7f5586977640 10 monclient: init 2024-03-29T11:17:29.853 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.874+0000 7f5586977640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:29.853 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.874+0000 7f5586977640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:29.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.876+0000 7f5586977640 10 monclient(hunting): picked mon.noname-a con 0x7f55800a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:29.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.876+0000 7f5586977640 10 monclient(hunting): picked mon.noname-b con 0x7f558013f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:29.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.876+0000 7f5586977640 10 monclient(hunting): picked mon.noname-c con 0x7f55801457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:29.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.876+0000 7f5586977640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5580140730 con 0x7f55800a9d70 2024-03-29T11:17:29.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.876+0000 7f5586977640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558013a390 con 0x7f558013f650 2024-03-29T11:17:29.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.876+0000 7f5586977640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558013b5d0 con 0x7f55801457f0 2024-03-29T11:17:29.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.876+0000 7f5586977640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:29.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.876+0000 7f5586977640 10 monclient(hunting): authenticate will time out at 1262.073373s 2024-03-29T11:17:29.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.877+0000 7f557ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f55800a9d70 0x7f5580139c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:47251/0 (socket says 172.21.15.157:47251) 2024-03-29T11:17:29.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557ffff640 1 -- 172.21.15.157:0/1785891259 learned_addr learned my addr 172.21.15.157:0/1785891259 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:29.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 10 client.?.objecter ms_handle_connect 0x7f55800a9d70 2024-03-29T11:17:29.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:29.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407554791 0 0) 0x7f5580140730 con 0x7f55800a9d70 2024-03-29T11:17:29.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:17:29.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:29.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:29.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:17:29.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f556c004eb0 con 0x7f55800a9d70 2024-03-29T11:17:29.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1985682818 0 0) 0x7f556c004eb0 con 0x7f55800a9d70 2024-03-29T11:17:29.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5580140730 con 0x7f55800a9d70 2024-03-29T11:17:29.859 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.878+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f55700017d0 con 0x7f55800a9d70 2024-03-29T11:17:29.860 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:29.860 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:29.860 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 10 monclient(hunting): dump: 2024-03-29T11:17:29.860 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:29.861 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1555356241 0 0) 0x7f5580140730 con 0x7f55800a9d70 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 >> v1:172.21.15.157:6790/0 conn(0x7f55801457f0 legacy=0x7f55801d7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 >> v1:172.21.15.195:6789/0 conn(0x7f558013f650 legacy=0x7f55801350d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 1 monclient: found mon.a 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55801391e0 con 0x7f55800a9d70 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 10 monclient: _finish_auth 0 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f557cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f5586977640 5 monclient: authenticate success, global_id 4449 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f5586977640 10 monclient: _renew_subs 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f5586977640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:17:29.862 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f5586977640 1 -- 172.21.15.157:0/1785891259 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5580138f00 con 0x7f55800a9d70 2024-03-29T11:17:29.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f5586977640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:29.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f5586977640 10 monclient: _renew_subs 2024-03-29T11:17:29.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f5586977640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:17:29.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f5586977640 1 -- 172.21.15.157:0/1785891259 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f558013bbc0 con 0x7f55800a9d70 2024-03-29T11:17:29.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.879+0000 7f5586977640 1 librados: init done 2024-03-29T11:17:29.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.880+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5570001930 con 0x7f55800a9d70 2024-03-29T11:17:29.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.880+0000 7f557cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:29.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.880+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f5570004a00 con 0x7f55800a9d70 2024-03-29T11:17:29.863 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.880+0000 7f557cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.880+0000 7f557cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.880+0000 7f557cff9640 10 monclient: dump: 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:29.864 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:29.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.881+0000 7f5586977640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:29.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.881+0000 7f557e7fc640 4 set_mon_vals no callback set 2024-03-29T11:17:29.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.882+0000 7f5586977640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:29.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.882+0000 7f5586977640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:17:29.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.882+0000 7f5586977640 1 -- 172.21.15.157:0/1785891259 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5580075f10 con 0x7f55800a9d70 2024-03-29T11:17:29.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.883+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f5570004b60 con 0x7f55800a9d70 2024-03-29T11:17:29.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.884+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f55700849c0 con 0x7f55800a9d70 2024-03-29T11:17:29.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.884+0000 7f557f7fe640 10 monclient: get_auth_request con 0x7f556c06dd60 auth_method 0 2024-03-29T11:17:29.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.884+0000 7f557cff9640 10 client.4449.objecter ms_dispatch 0x7f5580000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:29.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.884+0000 7f557cff9640 3 client.4449.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:29.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.884+0000 7f557cff9640 3 client.4449.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:29.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.884+0000 7f557cff9640 20 client.4449.objecter dump_active .. 0 homeless 2024-03-29T11:17:29.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.884+0000 7f557cff9640 10 client.4449.objecter ms_handle_connect 0x7f556c06dd60 2024-03-29T11:17:29.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.891+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55700502a0 con 0x7f55800a9d70 2024-03-29T11:17:29.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.891+0000 7f557cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:29.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:29.891+0000 7f557cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:30.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.219+0000 7f5586977640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:30.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.219+0000 7f5586977640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:30.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.219+0000 7f5586977640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:17:30.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.219+0000 7f5586977640 1 -- 172.21.15.157:0/1785891259 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f558004eee0 con 0x7f55800a9d70 2024-03-29T11:17:30.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.220+0000 7f557cff9640 1 -- 172.21.15.157:0/1785891259 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f55700521d0 con 0x7f55800a9d70 2024-03-29T11:17:30.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.220+0000 7f557cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:30.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.220+0000 7f557cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:30.199 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:17:30.199 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:30.201 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.224+0000 7f5586977640 10 librados: watch_flush enter 2024-03-29T11:17:30.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.224+0000 7f5586977640 10 librados: watch_flush exit 2024-03-29T11:17:30.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.224+0000 7f5586977640 20 client.4449.objecter shutdown clearing up homeless session... 2024-03-29T11:17:30.202 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.224+0000 7f5586977640 10 client.4449.objecter successfully canceled tick 2024-03-29T11:17:30.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.225+0000 7f5586977640 1 -- 172.21.15.157:0/1785891259 >> v1:172.21.15.157:6816/1747058247 conn(0x7f556c06dd60 legacy=0x7f556c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:30.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.225+0000 7f5586977640 10 monclient: shutdown 2024-03-29T11:17:30.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.225+0000 7f5586977640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:30.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.225+0000 7f5586977640 1 -- 172.21.15.157:0/1785891259 >> v1:172.21.15.157:6789/0 conn(0x7f55800a9d70 legacy=0x7f5580139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:30.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.225+0000 7f5586977640 1 -- 172.21.15.157:0/1785891259 shutdown_connections 2024-03-29T11:17:30.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.225+0000 7f5586977640 1 -- 172.21.15.157:0/1785891259 >> 172.21.15.157:0/1785891259 conn(0x7f5580066a00 msgr2=0x7f5580066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:30.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.226+0000 7f5586977640 1 -- 172.21.15.157:0/1785891259 shutdown_connections 2024-03-29T11:17:30.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.226+0000 7f5586977640 1 -- 172.21.15.157:0/1785891259 wait complete. 2024-03-29T11:17:30.203 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:30.226+0000 7f5586977640 1 librados: shutdown 2024-03-29T11:17:30.222 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:37.029 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:37.179 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.201+0000 7f2513227640 10 monclient: get_monmap_and_config 2024-03-29T11:17:37.179 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.201+0000 7f2513227640 10 monclient: build_initial_monmap 2024-03-29T11:17:37.179 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.201+0000 7f2513227640 10 monclient: monmap: 2024-03-29T11:17:37.179 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:37.179 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:37.179 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:17:37.202910+0000 2024-03-29T11:17:37.179 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:17:37.202910+0000 2024-03-29T11:17:37.180 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:37.180 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:37.180 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:37.180 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:17:37.180 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:17:37.180 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:37.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.203+0000 7f2513227640 1 Processor -- start 2024-03-29T11:17:37.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.203+0000 7f2513227640 1 -- start start 2024-03-29T11:17:37.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.203+0000 7f2513227640 10 monclient: init 2024-03-29T11:17:37.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.204+0000 7f2513227640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:37.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.204+0000 7f2513227640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:37.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.204+0000 7f2513227640 10 monclient(hunting): picked mon.noname-b con 0x7f250c0b0860 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:37.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.204+0000 7f2513227640 10 monclient(hunting): picked mon.noname-a con 0x7f250c13ef30 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:37.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.204+0000 7f2513227640 10 monclient(hunting): picked mon.noname-c con 0x7f250c0af8e0 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:37.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.205+0000 7f2510fc3640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f250c0b0860 0x7f250c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:39214/0 (socket says 172.21.15.157:39214) 2024-03-29T11:17:37.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.207+0000 7f2510fc3640 1 -- 172.21.15.157:0/4189946076 learned_addr learned my addr 172.21.15.157:0/4189946076 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:37.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.207+0000 7f2513227640 1 -- 172.21.15.157:0/4189946076 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c0b0c40 con 0x7f250c13ef30 2024-03-29T11:17:37.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.207+0000 7f2513227640 1 -- 172.21.15.157:0/4189946076 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c0aed10 con 0x7f250c0af8e0 2024-03-29T11:17:37.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.207+0000 7f2513227640 1 -- 172.21.15.157:0/4189946076 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c0afd50 con 0x7f250c0b0860 2024-03-29T11:17:37.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.207+0000 7f2513227640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:37.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.207+0000 7f2513227640 10 monclient(hunting): authenticate will time out at 1269.404351s 2024-03-29T11:17:37.187 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.209+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204252250 0 0) 0x7f250c0aed10 con 0x7f250c0af8e0 2024-03-29T11:17:37.187 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.209+0000 7f250b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:37.187 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.209+0000 7f250b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:37.187 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.209+0000 7f250b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:37.187 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.209+0000 7f250b7fe640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:17:37.188 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.209+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24fc003520 con 0x7f250c0af8e0 2024-03-29T11:17:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.210+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176017678 0 0) 0x7f250c0afd50 con 0x7f250c0b0860 2024-03-29T11:17:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.210+0000 7f250b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.210+0000 7f250b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.210+0000 7f250b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.210+0000 7f250b7fe640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:17:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.210+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f250c0aed10 con 0x7f250c0b0860 2024-03-29T11:17:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.210+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469189243 0 0) 0x7f250c0b0c40 con 0x7f250c13ef30 2024-03-29T11:17:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:37.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f250c0afd50 con 0x7f250c13ef30 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3721536311 0 0) 0x7f24fc003520 con 0x7f250c0af8e0 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 >> v1:172.21.15.157:6790/0 conn(0x7f250c0b0860 legacy=0x7f250c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 >> v1:172.21.15.157:6789/0 conn(0x7f250c13ef30 legacy=0x7f250c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c0ab1b0 con 0x7f250c0af8e0 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:37.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2504001a10 con 0x7f250c0af8e0 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f250b7fe640 10 monclient: dump: 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:37.195 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f2513227640 5 monclient: authenticate success, global_id 4435 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.211+0000 7f2513227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.213+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2504001c30 con 0x7f250c0af8e0 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.213+0000 7f250b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.213+0000 7f2513227640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.213+0000 7f2513227640 4 set_mon_vals no callback set 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.213+0000 7f250b7fe640 1 -- 172.21.15.157:0/4189946076 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2504001fc0 con 0x7f250c0af8e0 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.213+0000 7f250b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.213+0000 7f250b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.213+0000 7f250b7fe640 10 monclient: dump: 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:37.196 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.213+0000 7f2513227640 10 monclient: shutdown 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.214+0000 7f2513227640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.214+0000 7f2513227640 1 -- 172.21.15.157:0/4189946076 >> v1:172.21.15.195:6789/0 conn(0x7f250c0af8e0 legacy=0x7f250c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.214+0000 7f2513227640 1 -- 172.21.15.157:0/4189946076 shutdown_connections 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.214+0000 7f2513227640 1 -- 172.21.15.157:0/4189946076 >> 172.21.15.157:0/4189946076 conn(0x7f250c06b660 msgr2=0x7f250c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.214+0000 7f2513227640 1 -- 172.21.15.157:0/4189946076 shutdown_connections 2024-03-29T11:17:37.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.214+0000 7f2513227640 1 -- 172.21.15.157:0/4189946076 wait complete. 2024-03-29T11:17:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.215+0000 7f2513227640 10 monclient: build_initial_monmap 2024-03-29T11:17:37.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.215+0000 7f2513227640 10 monclient: monmap: 2024-03-29T11:17:37.205 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:37.205 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:37.205 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:17:37.205 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:17:37.205 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:37.205 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.216+0000 7f2513227640 1 librados: starting msgr at 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.216+0000 7f2513227640 1 librados: starting objecter 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.216+0000 7f2513227640 1 Processor -- start 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.216+0000 7f2513227640 1 -- start start 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.216+0000 7f2513227640 1 librados: setting wanted keys 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.216+0000 7f2513227640 1 librados: calling monclient init 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.216+0000 7f2513227640 10 monclient: init 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.217+0000 7f2513227640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.217+0000 7f2513227640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:37.206 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.217+0000 7f2513227640 10 monclient(hunting): picked mon.noname-c con 0x7f250c0af8e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.218+0000 7f2513227640 10 monclient(hunting): picked mon.noname-b con 0x7f250c0b0860 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.218+0000 7f2513227640 10 monclient(hunting): picked mon.noname-a con 0x7f250c13ef30 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.218+0000 7f2513227640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c070f10 con 0x7f250c13ef30 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.218+0000 7f2513227640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c06e9e0 con 0x7f250c0b0860 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.218+0000 7f2513227640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c14a210 con 0x7f250c0af8e0 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.218+0000 7f2513227640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.218+0000 7f2513227640 10 monclient(hunting): authenticate will time out at 1269.415451s 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.218+0000 7f2510fc3640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f250c0af8e0 0x7f250c071630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:42018/0 (socket says 172.21.15.157:42018) 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.218+0000 7f2510fc3640 1 -- 172.21.15.157:0/3534994363 learned_addr learned my addr 172.21.15.157:0/3534994363 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.219+0000 7f25097fa640 10 client.?.objecter ms_handle_connect 0x7f250c0af8e0 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.219+0000 7f25097fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.219+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042385619 0 0) 0x7f250c14a210 con 0x7f250c0af8e0 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.219+0000 7f25097fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:37.207 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.219+0000 7f25097fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.219+0000 7f25097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.219+0000 7f25097fa640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.220+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f0004eb0 con 0x7f250c0af8e0 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.220+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2111045233 0 0) 0x7f24f0004eb0 con 0x7f250c0af8e0 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.220+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f250c14a210 con 0x7f250c0af8e0 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.220+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f24f8005f80 con 0x7f250c0af8e0 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.220+0000 7f25097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.220+0000 7f25097fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.220+0000 7f25097fa640 10 monclient(hunting): dump: 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:37.208 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.220+0000 7f25097fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.221+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3768414785 0 0) 0x7f250c14a210 con 0x7f250c0af8e0 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.221+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 >> v1:172.21.15.195:6789/0 conn(0x7f250c0b0860 legacy=0x7f250c06ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.223+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 >> v1:172.21.15.157:6789/0 conn(0x7f250c13ef30 legacy=0x7f250c06f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.223+0000 7f25097fa640 10 monclient: _finish_hunting 0 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.223+0000 7f25097fa640 1 monclient: found mon.c 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.223+0000 7f25097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.223+0000 7f25097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.223+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c14a3a0 con 0x7f250c0af8e0 2024-03-29T11:17:37.209 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f25097fa640 10 monclient: _finish_auth 0 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f25097fa640 10 monclient: _check_auth_tickets 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f25097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f25097fa640 10 client.?.objecter ms_handle_connect 0x7f250c13ef30 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f2513227640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f2513227640 10 monclient: _renew_subs 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f2513227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f2513227640 1 -- 172.21.15.157:0/3534994363 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250c14a650 con 0x7f250c0af8e0 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f2513227640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f2513227640 10 monclient: _renew_subs 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f2513227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f2513227640 1 -- 172.21.15.157:0/3534994363 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f250c1e7310 con 0x7f250c0af8e0 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f2513227640 1 librados: init done 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.224+0000 7f25097fa640 10 client.4469.objecter resend_mon_ops 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.226+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24f8001250 con 0x7f250c0af8e0 2024-03-29T11:17:37.210 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.226+0000 7f25097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.226+0000 7f250affd640 4 set_mon_vals no callback set 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.226+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f24f80019a0 con 0x7f250c0af8e0 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.226+0000 7f25097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.226+0000 7f25097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.226+0000 7f25097fa640 10 monclient: dump: 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:37.211 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.227+0000 7f2513227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:37.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.227+0000 7f2513227640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:37.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.227+0000 7f2513227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:37.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.227+0000 7f2513227640 1 -- 172.21.15.157:0/3534994363 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ec005080 con 0x7f250c0af8e0 2024-03-29T11:17:37.212 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.228+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f24f8001b80 con 0x7f250c0af8e0 2024-03-29T11:17:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.228+0000 7f250bfff640 10 monclient: get_auth_request con 0x7f24f006db10 auth_method 0 2024-03-29T11:17:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.229+0000 7f25097fa640 10 client.4469.objecter ms_handle_connect 0x7f24f006db10 2024-03-29T11:17:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.229+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f24f8084c50 con 0x7f250c0af8e0 2024-03-29T11:17:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.229+0000 7f25097fa640 10 client.4469.objecter ms_dispatch 0x7f250c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.229+0000 7f25097fa640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.229+0000 7f25097fa640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:37.213 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.230+0000 7f25097fa640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T11:17:37.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.238+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24f8050530 con 0x7f250c0af8e0 2024-03-29T11:17:37.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.238+0000 7f25097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:37.215 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.238+0000 7f25097fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:37.537 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.560+0000 7f2513227640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:37.537 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.560+0000 7f2513227640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:37.537 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.560+0000 7f2513227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:37.537 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.560+0000 7f2513227640 1 -- 172.21.15.157:0/3534994363 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24ec005330 con 0x7f250c0af8e0 2024-03-29T11:17:37.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.561+0000 7f25097fa640 1 -- 172.21.15.157:0/3534994363 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f24f8052460 con 0x7f250c0af8e0 2024-03-29T11:17:37.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.561+0000 7f25097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:37.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.561+0000 7f25097fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:37.539 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:17:37.539 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:37.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.565+0000 7f2513227640 10 librados: watch_flush enter 2024-03-29T11:17:37.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.566+0000 7f2513227640 10 librados: watch_flush exit 2024-03-29T11:17:37.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.566+0000 7f2513227640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T11:17:37.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.566+0000 7f2513227640 10 client.4469.objecter successfully canceled tick 2024-03-29T11:17:37.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.566+0000 7f2513227640 1 -- 172.21.15.157:0/3534994363 >> v1:172.21.15.157:6816/1747058247 conn(0x7f24f006db10 legacy=0x7f24f006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:37.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.566+0000 7f2513227640 10 monclient: shutdown 2024-03-29T11:17:37.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.566+0000 7f2513227640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:37.544 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.566+0000 7f2513227640 1 -- 172.21.15.157:0/3534994363 >> v1:172.21.15.157:6790/0 conn(0x7f250c0af8e0 legacy=0x7f250c071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:37.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.567+0000 7f2513227640 1 -- 172.21.15.157:0/3534994363 shutdown_connections 2024-03-29T11:17:37.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.567+0000 7f2513227640 1 -- 172.21.15.157:0/3534994363 >> 172.21.15.157:0/3534994363 conn(0x7f250c06b660 msgr2=0x7f250c0ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:37.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.567+0000 7f2513227640 1 -- 172.21.15.157:0/3534994363 shutdown_connections 2024-03-29T11:17:37.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.567+0000 7f2513227640 1 -- 172.21.15.157:0/3534994363 wait complete. 2024-03-29T11:17:37.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:37.567+0000 7f2513227640 1 librados: shutdown 2024-03-29T11:17:37.562 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:44.369 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:44.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.553+0000 7f9aad435640 10 monclient: get_monmap_and_config 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.553+0000 7f9aad435640 10 monclient: build_initial_monmap 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.553+0000 7f9aad435640 10 monclient: monmap: 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:17:44.554702+0000 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:17:44.554702+0000 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:17:44.532 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:44.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.557+0000 7f9aad435640 1 Processor -- start 2024-03-29T11:17:44.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.558+0000 7f9aad435640 1 -- start start 2024-03-29T11:17:44.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.558+0000 7f9aad435640 10 monclient: init 2024-03-29T11:17:44.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.559+0000 7f9aad435640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:44.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.559+0000 7f9aad435640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:44.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.559+0000 7f9aad435640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa80a9ac0 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:44.536 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.559+0000 7f9aad435640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa813ee90 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:44.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.559+0000 7f9aad435640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa80af900 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:44.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.560+0000 7f9aad435640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa80afd10 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.560+0000 7f9aad435640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa80af6f0 con 0x7f9aa80af900 2024-03-29T11:17:44.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.560+0000 7f9aad435640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa80aac80 con 0x7f9aa813ee90 2024-03-29T11:17:44.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.560+0000 7f9aad435640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:44.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.560+0000 7f9aad435640 10 monclient(hunting): authenticate will time out at 1276.757506s 2024-03-29T11:17:44.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.563+0000 7f9aa67fc640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f9aa813ee90 0x7f9aa813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:35965/0 (socket says 172.21.15.157:35965) 2024-03-29T11:17:44.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.563+0000 7f9aa67fc640 1 -- 172.21.15.157:0/3924019744 learned_addr learned my addr 172.21.15.157:0/3924019744 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:44.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.566+0000 7f9aa5ffb640 1 -- 172.21.15.157:0/3924019744 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834249627 0 0) 0x7f9aa80aac80 con 0x7f9aa813ee90 2024-03-29T11:17:44.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.566+0000 7f9aa5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:44.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.566+0000 7f9aa5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:44.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.566+0000 7f9aa5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:44.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.566+0000 7f9aa5ffb640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:17:44.545 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.566+0000 7f9aa5ffb640 1 -- 172.21.15.157:0/3924019744 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a90003520 con 0x7f9aa813ee90 2024-03-29T11:17:44.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 1 -- 172.21.15.157:0/3924019744 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3330675188 0 0) 0x7f9a90003520 con 0x7f9aa813ee90 2024-03-29T11:17:44.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 1 -- 172.21.15.157:0/3924019744 >> v1:172.21.15.195:6789/0 conn(0x7f9aa80af900 legacy=0x7f9aa8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:44.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 1 -- 172.21.15.157:0/3924019744 >> v1:172.21.15.157:6789/0 conn(0x7f9aa80a9ac0 legacy=0x7f9aa80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:44.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:44.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:17:44.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:44.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:17:44.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 1 -- 172.21.15.157:0/3924019744 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa80aaf60 con 0x7f9aa813ee90 2024-03-29T11:17:44.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:44.546 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:44.547 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:44.547 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 1 -- 172.21.15.157:0/3924019744 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9aa0001a10 con 0x7f9aa813ee90 2024-03-29T11:17:44.547 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:44.547 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:44.547 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aa5ffb640 10 monclient: dump: 2024-03-29T11:17:44.547 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aad435640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.567+0000 7f9aad435640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.568+0000 7f9aa5ffb640 1 -- 172.21.15.157:0/3924019744 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aa0001b70 con 0x7f9aa813ee90 2024-03-29T11:17:44.548 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.568+0000 7f9aa5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.568+0000 7f9aa5ffb640 1 -- 172.21.15.157:0/3924019744 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9aa0001fc0 con 0x7f9aa813ee90 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.568+0000 7f9aa5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.568+0000 7f9aa5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.568+0000 7f9aa5ffb640 10 monclient: dump: 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.568+0000 7f9aad435640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:44.549 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.568+0000 7f9aad435640 4 set_mon_vals no callback set 2024-03-29T11:17:44.550 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.569+0000 7f9aad435640 10 monclient: shutdown 2024-03-29T11:17:44.550 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.569+0000 7f9aad435640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:44.550 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.569+0000 7f9aad435640 1 -- 172.21.15.157:0/3924019744 >> v1:172.21.15.157:6790/0 conn(0x7f9aa813ee90 legacy=0x7f9aa813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:44.550 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.569+0000 7f9aad435640 1 -- 172.21.15.157:0/3924019744 shutdown_connections 2024-03-29T11:17:44.550 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.569+0000 7f9aad435640 1 -- 172.21.15.157:0/3924019744 >> 172.21.15.157:0/3924019744 conn(0x7f9aa80b35e0 msgr2=0x7f9aa80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:44.550 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.571+0000 7f9aad435640 1 -- 172.21.15.157:0/3924019744 shutdown_connections 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.571+0000 7f9aad435640 1 -- 172.21.15.157:0/3924019744 wait complete. 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.573+0000 7f9aad435640 10 monclient: build_initial_monmap 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.573+0000 7f9aad435640 10 monclient: monmap: 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:17:44.551 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:44.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.574+0000 7f9aad435640 1 librados: starting msgr at 2024-03-29T11:17:44.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.574+0000 7f9aad435640 1 librados: starting objecter 2024-03-29T11:17:44.552 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.575+0000 7f9aad435640 1 Processor -- start 2024-03-29T11:17:44.553 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.576+0000 7f9aad435640 1 -- start start 2024-03-29T11:17:44.554 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.576+0000 7f9aad435640 1 librados: setting wanted keys 2024-03-29T11:17:44.554 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.576+0000 7f9aad435640 1 librados: calling monclient init 2024-03-29T11:17:44.554 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.576+0000 7f9aad435640 10 monclient: init 2024-03-29T11:17:44.555 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.577+0000 7f9aad435640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:44.555 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.577+0000 7f9aad435640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:44.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.577+0000 7f9aad435640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa80a9ac0 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:44.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.577+0000 7f9aad435640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa80af900 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:44.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.577+0000 7f9aad435640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa813ee90 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:44.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.577+0000 7f9aad435640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa8141ca0 con 0x7f9aa80af900 2024-03-29T11:17:44.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.578+0000 7f9aad435640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa8140bb0 con 0x7f9aa813ee90 2024-03-29T11:17:44.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.578+0000 7f9aad435640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa8140730 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.578+0000 7f9aad435640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:44.556 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.578+0000 7f9aad435640 10 monclient(hunting): authenticate will time out at 1276.774890s 2024-03-29T11:17:44.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.584+0000 7f9aa6ffd640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f9aa80a9ac0 0x7f9aa8140020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:36081/0 (socket says 172.21.15.157:36081) 2024-03-29T11:17:44.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.584+0000 7f9aa6ffd640 1 -- 172.21.15.157:0/1498891708 learned_addr learned my addr 172.21.15.157:0/1498891708 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:44.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.584+0000 7f9a8f7fe640 10 client.?.objecter ms_handle_connect 0x7f9aa80a9ac0 2024-03-29T11:17:44.561 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.584+0000 7f9a8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:44.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087503862 0 0) 0x7f9aa8140730 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a94004eb0 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 10 client.?.objecter ms_handle_connect 0x7f9aa813ee90 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602864433 0 0) 0x7f9aa8140bb0 con 0x7f9aa813ee90 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.587+0000 7f9a8f7fe640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa8140730 con 0x7f9aa813ee90 2024-03-29T11:17:44.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3171945198 0 0) 0x7f9a94004eb0 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aa8140bb0 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9a9c001a30 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 10 monclient(hunting): dump: 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:44.569 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1636656421 0 0) 0x7f9aa8140bb0 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 >> v1:172.21.15.195:6789/0 conn(0x7f9aa813ee90 legacy=0x7f9aa81d7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.588+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 >> v1:172.21.15.157:6789/0 conn(0x7f9aa80af900 legacy=0x7f9aa81d7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9a8f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9a8f7fe640 1 monclient: found mon.c 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9a8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9a8f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa81408c0 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9a8f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9a8f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9a8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9a8f7fe640 10 client.?.objecter ms_handle_connect 0x7f9aa80af900 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9a8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:44.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9aad435640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:17:44.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9aad435640 10 monclient: _renew_subs 2024-03-29T11:17:44.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9aad435640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:44.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9aad435640 1 -- 172.21.15.157:0/1498891708 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa81423a0 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9aad435640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:44.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9aad435640 10 monclient: _renew_subs 2024-03-29T11:17:44.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9aad435640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:44.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9aad435640 1 -- 172.21.15.157:0/1498891708 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa81e3390 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.589+0000 7f9aad435640 1 librados: init done 2024-03-29T11:17:44.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.590+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a9c001c70 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.590+0000 7f9a8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:44.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.590+0000 7f9aa57fa640 4 set_mon_vals no callback set 2024-03-29T11:17:44.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.591+0000 7f9aad435640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:44.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.591+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f9a9c003a60 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.591+0000 7f9a8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.591+0000 7f9a8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.591+0000 7f9a8f7fe640 10 monclient: dump: 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.591+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f9a9c003c40 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.591+0000 7f9aad435640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:44.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.592+0000 7f9aad435640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:44.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.592+0000 7f9aad435640 1 -- 172.21.15.157:0/1498891708 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a88005080 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.592+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f9a9c084b20 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.592+0000 7f9a8f7fe640 10 client.4475.objecter ms_dispatch 0x7f9aa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:44.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.592+0000 7f9a8f7fe640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:44.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.592+0000 7f9a8f7fe640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:44.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.592+0000 7f9a8f7fe640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:17:44.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.595+0000 7f9aa67fc640 10 monclient: get_auth_request con 0x7f9a9406db40 auth_method 0 2024-03-29T11:17:44.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.596+0000 7f9a8f7fe640 10 client.4475.objecter ms_handle_connect 0x7f9a9406db40 2024-03-29T11:17:44.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.604+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a9c0504e0 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.604+0000 7f9a8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:44.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.604+0000 7f9a8f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:44.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.920+0000 7f9aad435640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:44.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.920+0000 7f9aad435640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:44.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.920+0000 7f9aad435640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:44.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.920+0000 7f9aad435640 1 -- 172.21.15.157:0/1498891708 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a88005330 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.925+0000 7f9a8f7fe640 1 -- 172.21.15.157:0/1498891708 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f9a9c052410 con 0x7f9aa80a9ac0 2024-03-29T11:17:44.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.925+0000 7f9a8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:44.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.925+0000 7f9a8f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:44.903 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:17:44.903 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:44.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.928+0000 7f9aad435640 10 librados: watch_flush enter 2024-03-29T11:17:44.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.928+0000 7f9aad435640 10 librados: watch_flush exit 2024-03-29T11:17:44.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.928+0000 7f9aad435640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:17:44.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.928+0000 7f9aad435640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:17:44.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.928+0000 7f9aad435640 1 -- 172.21.15.157:0/1498891708 >> v1:172.21.15.157:6816/1747058247 conn(0x7f9a9406db40 legacy=0x7f9a94070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:44.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.928+0000 7f9aad435640 10 monclient: shutdown 2024-03-29T11:17:44.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.929+0000 7f9aad435640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:44.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.929+0000 7f9aad435640 1 -- 172.21.15.157:0/1498891708 >> v1:172.21.15.157:6790/0 conn(0x7f9aa80a9ac0 legacy=0x7f9aa8140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:44.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.929+0000 7f9aad435640 1 -- 172.21.15.157:0/1498891708 shutdown_connections 2024-03-29T11:17:44.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.929+0000 7f9aad435640 1 -- 172.21.15.157:0/1498891708 >> 172.21.15.157:0/1498891708 conn(0x7f9aa80b35e0 msgr2=0x7f9aa80ad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:44.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.929+0000 7f9aad435640 1 -- 172.21.15.157:0/1498891708 shutdown_connections 2024-03-29T11:17:44.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.929+0000 7f9aad435640 1 -- 172.21.15.157:0/1498891708 wait complete. 2024-03-29T11:17:44.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:44.929+0000 7f9aad435640 1 librados: shutdown 2024-03-29T11:17:44.926 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:51.733 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:51.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.909+0000 7ff3a2146640 10 monclient: get_monmap_and_config 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.909+0000 7ff3a2146640 10 monclient: build_initial_monmap 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.909+0000 7ff3a2146640 10 monclient: monmap: 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:17:51.910531+0000 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:17:51.910531+0000 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:17:51.887 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:51.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.910+0000 7ff3a2146640 1 Processor -- start 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.910+0000 7ff3a2146640 1 -- start start 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.910+0000 7ff3a2146640 10 monclient: init 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff3a2146640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff3a2146640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff3a2146640 10 monclient(hunting): picked mon.noname-a con 0x7ff39c0b0500 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff3a2146640 10 monclient(hunting): picked mon.noname-b con 0x7ff39c0b08e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff3a2146640 10 monclient(hunting): picked mon.noname-c con 0x7ff39c0aeca0 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff3a2146640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c140620 con 0x7ff39c0b0500 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff3a2146640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c13daf0 con 0x7ff39c0aeca0 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff3a2146640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c0ae180 con 0x7ff39c0b08e0 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff3a2146640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff3a2146640 10 monclient(hunting): authenticate will time out at 1284.108464s 2024-03-29T11:17:51.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff39affd640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ff39c0b08e0 0x7ff39c13d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:41134/0 (socket says 172.21.15.157:41134) 2024-03-29T11:17:51.890 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.911+0000 7ff39affd640 1 -- 172.21.15.157:0/3190711021 learned_addr learned my addr 172.21.15.157:0/3190711021 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632074690 0 0) 0x7ff39c0ae180 con 0x7ff39c0b08e0 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff384003520 con 0x7ff39c0b08e0 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879181282 0 0) 0x7ff39c140620 con 0x7ff39c0b0500 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c0ae180 con 0x7ff39c0b0500 2024-03-29T11:17:51.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888790491 0 0) 0x7ff39c0ae180 con 0x7ff39c0b0500 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 >> v1:172.21.15.157:6790/0 conn(0x7ff39c0b08e0 legacy=0x7ff39c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 >> v1:172.21.15.195:6789/0 conn(0x7ff39c0aeca0 legacy=0x7ff39c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c072510 con 0x7ff39c0b0500 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff390001670 con 0x7ff39c0b0500 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff39a7fc640 10 monclient: dump: 2024-03-29T11:17:51.895 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff3a2146640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:17:51.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.912+0000 7ff3a2146640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.913+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3900017d0 con 0x7ff39c0b0500 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.913+0000 7ff39a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.913+0000 7ff39a7fc640 1 -- 172.21.15.157:0/3190711021 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff390001c20 con 0x7ff39c0b0500 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.913+0000 7ff39a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.913+0000 7ff39a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.913+0000 7ff39a7fc640 10 monclient: dump: 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:51.897 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.913+0000 7ff3a2146640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.913+0000 7ff3a2146640 4 set_mon_vals no callback set 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.914+0000 7ff3a2146640 10 monclient: shutdown 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.914+0000 7ff3a2146640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.914+0000 7ff3a2146640 1 -- 172.21.15.157:0/3190711021 >> v1:172.21.15.157:6789/0 conn(0x7ff39c0b0500 legacy=0x7ff39c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.914+0000 7ff3a2146640 1 -- 172.21.15.157:0/3190711021 shutdown_connections 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.914+0000 7ff3a2146640 1 -- 172.21.15.157:0/3190711021 >> 172.21.15.157:0/3190711021 conn(0x7ff39c06b660 msgr2=0x7ff39c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.914+0000 7ff3a2146640 1 -- 172.21.15.157:0/3190711021 shutdown_connections 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.914+0000 7ff3a2146640 1 -- 172.21.15.157:0/3190711021 wait complete. 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.915+0000 7ff3a2146640 10 monclient: build_initial_monmap 2024-03-29T11:17:51.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.915+0000 7ff3a2146640 10 monclient: monmap: 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.915+0000 7ff3a2146640 1 librados: starting msgr at 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.915+0000 7ff3a2146640 1 librados: starting objecter 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.915+0000 7ff3a2146640 1 Processor -- start 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.916+0000 7ff3a2146640 1 -- start start 2024-03-29T11:17:51.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.916+0000 7ff3a2146640 1 librados: setting wanted keys 2024-03-29T11:17:51.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.916+0000 7ff3a2146640 1 librados: calling monclient init 2024-03-29T11:17:51.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.916+0000 7ff3a2146640 10 monclient: init 2024-03-29T11:17:51.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff3a2146640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:51.900 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff3a2146640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:51.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff3a2146640 10 monclient(hunting): picked mon.noname-c con 0x7ff39c0aeca0 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:51.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff3a2146640 10 monclient(hunting): picked mon.noname-b con 0x7ff39c0b0500 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:51.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff3a2146640 10 monclient(hunting): picked mon.noname-a con 0x7ff39c14a020 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:51.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff3a2146640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c13d2c0 con 0x7ff39c14a020 2024-03-29T11:17:51.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff3a2146640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c14f970 con 0x7ff39c0b0500 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff3a2146640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c14fbe0 con 0x7ff39c0aeca0 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff3a2146640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff3a2146640 10 monclient(hunting): authenticate will time out at 1284.114172s 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff39b7fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ff39c0aeca0 0x7ff39c150370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:41480/0 (socket says 172.21.15.157:41480) 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff39b7fe640 1 -- 172.21.15.157:0/1642204751 learned_addr learned my addr 172.21.15.157:0/1642204751 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff383fff640 10 client.?.objecter ms_handle_connect 0x7ff39c0aeca0 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.917+0000 7ff383fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595549384 0 0) 0x7ff39c14fbe0 con 0x7ff39c0aeca0 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff388004eb0 con 0x7ff39c0aeca0 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 client.?.objecter ms_handle_connect 0x7ff39c14a020 2024-03-29T11:17:51.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 client.?.objecter ms_handle_connect 0x7ff39c0b0500 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332260458 0 0) 0x7ff39c13d2c0 con 0x7ff39c14a020 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.918+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c14fbe0 con 0x7ff39c14a020 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.919+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3756184786 0 0) 0x7ff388004eb0 con 0x7ff39c0aeca0 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.919+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff39c13d2c0 con 0x7ff39c0aeca0 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.919+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3513720340 0 0) 0x7ff39c14fbe0 con 0x7ff39c14a020 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.919+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff388004eb0 con 0x7ff39c14a020 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.919+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff390001630 con 0x7ff39c0aeca0 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.919+0000 7ff383fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:51.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.919+0000 7ff383fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.919+0000 7ff383fff640 10 monclient(hunting): dump: 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff394001710 con 0x7ff39c14a020 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:51.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 10 monclient(hunting): dump: 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1143187075 0 0) 0x7ff39c13d2c0 con 0x7ff39c0aeca0 2024-03-29T11:17:51.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 >> v1:172.21.15.195:6789/0 conn(0x7ff39c0b0500 legacy=0x7ff39c14da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 >> v1:172.21.15.157:6789/0 conn(0x7ff39c14a020 legacy=0x7ff39c14e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 1 monclient: found mon.c 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c13d9c0 con 0x7ff39c0aeca0 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff383fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff3a2146640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff3a2146640 10 monclient: _renew_subs 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff3a2146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.920+0000 7ff3a2146640 1 -- 172.21.15.157:0/1642204751 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39c151160 con 0x7ff39c0aeca0 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff3a2146640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff3a2146640 10 monclient: _renew_subs 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff3a2146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:51.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff3a2146640 1 -- 172.21.15.157:0/1642204751 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39c14ea40 con 0x7ff39c0aeca0 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff3a2146640 1 librados: init done 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3900039c0 con 0x7ff39c0aeca0 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff383fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff3997fa640 4 set_mon_vals no callback set 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff390003d10 con 0x7ff39c0aeca0 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff383fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff383fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.921+0000 7ff383fff640 10 monclient: dump: 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:51.912 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.922+0000 7ff3a2146640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.922+0000 7ff3a2146640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.922+0000 7ff3a2146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.922+0000 7ff3a2146640 1 -- 172.21.15.157:0/1642204751 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39c0ad0c0 con 0x7ff39c0aeca0 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.923+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7ff390003e70 con 0x7ff39c0aeca0 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.924+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7ff390084b20 con 0x7ff39c0aeca0 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.924+0000 7ff383fff640 10 client.4481.objecter ms_dispatch 0x7ff39c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.924+0000 7ff383fff640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.924+0000 7ff383fff640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.924+0000 7ff383fff640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.924+0000 7ff39affd640 10 monclient: get_auth_request con 0x7ff38806e0b0 auth_method 0 2024-03-29T11:17:51.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.924+0000 7ff383fff640 10 client.4481.objecter ms_handle_connect 0x7ff38806e0b0 2024-03-29T11:17:51.914 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.933+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff390050400 con 0x7ff39c0aeca0 2024-03-29T11:17:51.914 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.933+0000 7ff383fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:51.914 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:51.933+0000 7ff383fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:52.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.312+0000 7ff3a2146640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:52.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.312+0000 7ff3a2146640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:52.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.312+0000 7ff3a2146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:17:52.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.312+0000 7ff3a2146640 1 -- 172.21.15.157:0/1642204751 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff39c075f10 con 0x7ff39c0aeca0 2024-03-29T11:17:52.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.313+0000 7ff383fff640 1 -- 172.21.15.157:0/1642204751 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7ff390052330 con 0x7ff39c0aeca0 2024-03-29T11:17:52.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.314+0000 7ff383fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:52.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.314+0000 7ff383fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:52.291 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:17:52.291 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:52.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.318+0000 7ff3a2146640 10 librados: watch_flush enter 2024-03-29T11:17:52.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.318+0000 7ff3a2146640 10 librados: watch_flush exit 2024-03-29T11:17:52.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.318+0000 7ff3a2146640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:17:52.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.318+0000 7ff3a2146640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:17:52.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.318+0000 7ff3a2146640 1 -- 172.21.15.157:0/1642204751 >> v1:172.21.15.157:6816/1747058247 conn(0x7ff38806e0b0 legacy=0x7ff388070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:52.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.318+0000 7ff3a2146640 10 monclient: shutdown 2024-03-29T11:17:52.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.318+0000 7ff3a2146640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:52.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.318+0000 7ff3a2146640 1 -- 172.21.15.157:0/1642204751 >> v1:172.21.15.157:6790/0 conn(0x7ff39c0aeca0 legacy=0x7ff39c150370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:52.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.318+0000 7ff3a2146640 1 -- 172.21.15.157:0/1642204751 shutdown_connections 2024-03-29T11:17:52.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.318+0000 7ff3a2146640 1 -- 172.21.15.157:0/1642204751 >> 172.21.15.157:0/1642204751 conn(0x7ff39c06b660 msgr2=0x7ff39c13e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:52.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.319+0000 7ff3a2146640 1 -- 172.21.15.157:0/1642204751 shutdown_connections 2024-03-29T11:17:52.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.319+0000 7ff3a2146640 1 -- 172.21.15.157:0/1642204751 wait complete. 2024-03-29T11:17:52.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:52.319+0000 7ff3a2146640 1 librados: shutdown 2024-03-29T11:17:52.313 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:59.120 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:59.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.288+0000 7f2e75020640 10 monclient: get_monmap_and_config 2024-03-29T11:17:59.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.288+0000 7f2e75020640 10 monclient: build_initial_monmap 2024-03-29T11:17:59.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.288+0000 7f2e75020640 10 monclient: monmap: 2024-03-29T11:17:59.265 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:59.265 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:59.266 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:17:59.289127+0000 2024-03-29T11:17:59.266 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:17:59.289127+0000 2024-03-29T11:17:59.266 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:59.266 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:59.266 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:59.266 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:17:59.266 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:17:59.266 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:59.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.290+0000 7f2e75020640 1 Processor -- start 2024-03-29T11:17:59.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.290+0000 7f2e75020640 1 -- start start 2024-03-29T11:17:59.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.290+0000 7f2e75020640 10 monclient: init 2024-03-29T11:17:59.273 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.291+0000 7f2e75020640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:59.273 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.291+0000 7f2e75020640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:59.273 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.291+0000 7f2e75020640 10 monclient(hunting): picked mon.noname-b con 0x7f2e700b4330 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:59.273 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.291+0000 7f2e75020640 10 monclient(hunting): picked mon.noname-a con 0x7f2e701470b0 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:59.273 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.291+0000 7f2e75020640 10 monclient(hunting): picked mon.noname-c con 0x7f2e701439b0 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.291+0000 7f2e75020640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7006f2a0 con 0x7f2e701470b0 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.291+0000 7f2e75020640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e70146b90 con 0x7f2e701439b0 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.291+0000 7f2e75020640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e700b2ca0 con 0x7f2e700b4330 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.291+0000 7f2e75020640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.291+0000 7f2e75020640 10 monclient(hunting): authenticate will time out at 1291.488683s 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.294+0000 7f2e6f59e640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2e701439b0 0x7f2e70151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:44599/0 (socket says 172.21.15.157:44599) 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.294+0000 7f2e6f59e640 1 -- 172.21.15.157:0/2804425612 learned_addr learned my addr 172.21.15.157:0/2804425612 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.295+0000 7f2e6dd9b640 1 -- 172.21.15.157:0/2804425612 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830124869 0 0) 0x7f2e70146b90 con 0x7f2e701439b0 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.295+0000 7f2e6dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.295+0000 7f2e6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.295+0000 7f2e6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.295+0000 7f2e6dd9b640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:17:59.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.295+0000 7f2e6dd9b640 1 -- 172.21.15.157:0/2804425612 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e58003520 con 0x7f2e701439b0 2024-03-29T11:17:59.275 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.295+0000 7f2e6dd9b640 1 -- 172.21.15.157:0/2804425612 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1841154155 0 0) 0x7f2e58003520 con 0x7f2e701439b0 2024-03-29T11:17:59.275 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.295+0000 7f2e6dd9b640 1 -- 172.21.15.157:0/2804425612 >> v1:172.21.15.157:6790/0 conn(0x7f2e700b4330 legacy=0x7f2e700b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:59.275 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.298+0000 7f2e6dd9b640 1 -- 172.21.15.157:0/2804425612 >> v1:172.21.15.157:6789/0 conn(0x7f2e701470b0 legacy=0x7f2e700b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:59.280 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:17:59.280 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 1 monclient: found mon.noname-c 2024-03-29T11:17:59.280 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 1 -- 172.21.15.157:0/2804425612 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7006e0c0 con 0x7f2e701439b0 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 1 -- 172.21.15.157:0/2804425612 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2e64001690 con 0x7f2e701439b0 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e6dd9b640 10 monclient: dump: 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:59.281 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:59.282 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:59.282 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:59.282 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:59.282 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:59.282 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:59.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e75020640 5 monclient: authenticate success, global_id 4444 2024-03-29T11:17:59.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.302+0000 7f2e75020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:59.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.303+0000 7f2e6dd9b640 1 -- 172.21.15.157:0/2804425612 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e64001860 con 0x7f2e701439b0 2024-03-29T11:17:59.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.303+0000 7f2e6dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:59.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.303+0000 7f2e75020640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:59.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.303+0000 7f2e75020640 4 set_mon_vals no callback set 2024-03-29T11:17:59.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.304+0000 7f2e75020640 10 monclient: shutdown 2024-03-29T11:17:59.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.304+0000 7f2e75020640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:59.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.304+0000 7f2e75020640 1 -- 172.21.15.157:0/2804425612 >> v1:172.21.15.195:6789/0 conn(0x7f2e701439b0 legacy=0x7f2e70151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:59.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.305+0000 7f2e75020640 1 -- 172.21.15.157:0/2804425612 shutdown_connections 2024-03-29T11:17:59.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.305+0000 7f2e75020640 1 -- 172.21.15.157:0/2804425612 >> 172.21.15.157:0/2804425612 conn(0x7f2e70066a00 msgr2=0x7f2e70066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:59.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.306+0000 7f2e75020640 1 -- 172.21.15.157:0/2804425612 shutdown_connections 2024-03-29T11:17:59.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.306+0000 7f2e75020640 1 -- 172.21.15.157:0/2804425612 wait complete. 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.307+0000 7f2e75020640 10 monclient: build_initial_monmap 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.307+0000 7f2e75020640 10 monclient: monmap: 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:17:59.285 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:17:59.286 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:59.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.307+0000 7f2e75020640 1 librados: starting msgr at 2024-03-29T11:17:59.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.307+0000 7f2e75020640 1 librados: starting objecter 2024-03-29T11:17:59.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.308+0000 7f2e75020640 1 Processor -- start 2024-03-29T11:17:59.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.311+0000 7f2e75020640 1 -- start start 2024-03-29T11:17:59.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.311+0000 7f2e75020640 1 librados: setting wanted keys 2024-03-29T11:17:59.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.311+0000 7f2e75020640 1 librados: calling monclient init 2024-03-29T11:17:59.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.311+0000 7f2e75020640 10 monclient: init 2024-03-29T11:17:59.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.312+0000 7f2e75020640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:59.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.312+0000 7f2e75020640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:59.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.312+0000 7f2e75020640 10 monclient(hunting): picked mon.noname-c con 0x7f2e700b4330 addr v1:172.21.15.157:6790/0 2024-03-29T11:17:59.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.314+0000 7f2e75020640 10 monclient(hunting): picked mon.noname-b con 0x7f2e701439b0 addr v1:172.21.15.195:6789/0 2024-03-29T11:17:59.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.315+0000 7f2e75020640 10 monclient(hunting): picked mon.noname-a con 0x7f2e701470b0 addr v1:172.21.15.157:6789/0 2024-03-29T11:17:59.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e75020640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7011fce0 con 0x7f2e701470b0 2024-03-29T11:17:59.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e6e59c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2e701439b0 0x7f2e7013ff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:44719/0 (socket says 172.21.15.157:44719) 2024-03-29T11:17:59.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e6e59c640 1 -- 172.21.15.157:0/1751497753 learned_addr learned my addr 172.21.15.157:0/1751497753 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:17:59.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e701dee00 con 0x7f2e701439b0 2024-03-29T11:17:59.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7011f770 con 0x7f2e700b4330 2024-03-29T11:17:59.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e75020640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:59.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e75020640 10 monclient(hunting): authenticate will time out at 1291.513389s 2024-03-29T11:17:59.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e577fe640 10 client.?.objecter ms_handle_connect 0x7f2e701439b0 2024-03-29T11:17:59.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:59.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205344378 0 0) 0x7f2e701dee00 con 0x7f2e701439b0 2024-03-29T11:17:59.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:59.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:59.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:59.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e577fe640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:17:59.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.316+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c004eb0 con 0x7f2e701439b0 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.318+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335507714 0 0) 0x7f2e5c004eb0 con 0x7f2e701439b0 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.318+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e701dee00 con 0x7f2e701439b0 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.319+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2e68001ce0 con 0x7f2e701439b0 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.319+0000 7f2e577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.319+0000 7f2e577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.319+0000 7f2e577fe640 10 monclient(hunting): dump: 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:59.296 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:59.297 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:59.297 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:59.297 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:59.297 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:59.297 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:59.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.319+0000 7f2e577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:59.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.319+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2163408822 0 0) 0x7f2e701dee00 con 0x7f2e701439b0 2024-03-29T11:17:59.297 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.319+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 >> v1:172.21.15.157:6790/0 conn(0x7f2e700b4330 legacy=0x7f2e701de6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.319+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 >> v1:172.21.15.157:6789/0 conn(0x7f2e701470b0 legacy=0x7f2e701e6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 1 monclient: found mon.b 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7011f900 con 0x7f2e701439b0 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e68001e40 con 0x7f2e701439b0 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:59.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2e68003dc0 con 0x7f2e701439b0 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e6cd99640 4 set_mon_vals no callback set 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.320+0000 7f2e577fe640 10 monclient: dump: 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:17:59.300 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:17:59.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.321+0000 7f2e75020640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:17:59.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.321+0000 7f2e75020640 10 monclient: _renew_subs 2024-03-29T11:17:59.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.321+0000 7f2e75020640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:59.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.321+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e70140d30 con 0x7f2e701439b0 2024-03-29T11:17:59.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.321+0000 7f2e75020640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:59.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.321+0000 7f2e75020640 10 monclient: _renew_subs 2024-03-29T11:17:59.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.321+0000 7f2e75020640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:59.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.321+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e701410d0 con 0x7f2e701439b0 2024-03-29T11:17:59.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.321+0000 7f2e75020640 1 librados: init done 2024-03-29T11:17:59.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.324+0000 7f2e75020640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:59.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.324+0000 7f2e75020640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:59.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.324+0000 7f2e75020640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:59.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.324+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7011fb50 con 0x7f2e701439b0 2024-03-29T11:17:59.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.325+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f2e68003fa0 con 0x7f2e701439b0 2024-03-29T11:17:59.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.326+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f2e68084f60 con 0x7f2e701439b0 2024-03-29T11:17:59.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.326+0000 7f2e577fe640 10 client.4447.objecter ms_dispatch 0x7f2e70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:59.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.326+0000 7f2e577fe640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:59.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.326+0000 7f2e577fe640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:59.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.326+0000 7f2e6ed9d640 10 monclient: get_auth_request con 0x7f2e5c06db80 auth_method 0 2024-03-29T11:17:59.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.327+0000 7f2e577fe640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:17:59.305 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.328+0000 7f2e577fe640 10 client.4447.objecter ms_handle_connect 0x7f2e5c06db80 2024-03-29T11:17:59.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.335+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e68050840 con 0x7f2e701439b0 2024-03-29T11:17:59.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.335+0000 7f2e577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:59.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.335+0000 7f2e577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:59.643 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.665+0000 7f2e75020640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:59.643 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.665+0000 7f2e75020640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:59.643 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.665+0000 7f2e75020640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:17:59.643 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.665+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e701d1b00 con 0x7f2e701439b0 2024-03-29T11:17:59.644 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.666+0000 7f2e577fe640 1 -- 172.21.15.157:0/1751497753 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f2e68052770 con 0x7f2e701439b0 2024-03-29T11:17:59.644 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.666+0000 7f2e577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:59.644 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.666+0000 7f2e577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:59.644 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:17:59.644 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:59.647 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.669+0000 7f2e75020640 10 librados: watch_flush enter 2024-03-29T11:17:59.647 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.670+0000 7f2e75020640 10 librados: watch_flush exit 2024-03-29T11:17:59.647 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.670+0000 7f2e75020640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:17:59.647 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.670+0000 7f2e75020640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:17:59.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.670+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 >> v1:172.21.15.157:6816/1747058247 conn(0x7f2e5c06db80 legacy=0x7f2e5c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:59.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.670+0000 7f2e75020640 10 monclient: shutdown 2024-03-29T11:17:59.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.670+0000 7f2e75020640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:59.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.670+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 >> v1:172.21.15.195:6789/0 conn(0x7f2e701439b0 legacy=0x7f2e7013ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:59.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.671+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 shutdown_connections 2024-03-29T11:17:59.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.671+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 >> 172.21.15.157:0/1751497753 conn(0x7f2e70066a00 msgr2=0x7f2e70066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:59.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.671+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 shutdown_connections 2024-03-29T11:17:59.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.671+0000 7f2e75020640 1 -- 172.21.15.157:0/1751497753 wait complete. 2024-03-29T11:17:59.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:17:59.671+0000 7f2e75020640 1 librados: shutdown 2024-03-29T11:17:59.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:06.474 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:06.620 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.642+0000 7fc9a359e640 10 monclient: get_monmap_and_config 2024-03-29T11:18:06.620 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.642+0000 7fc9a359e640 10 monclient: build_initial_monmap 2024-03-29T11:18:06.620 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.642+0000 7fc9a359e640 10 monclient: monmap: 2024-03-29T11:18:06.620 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:06.620 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:06.620 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:18:06.643704+0000 2024-03-29T11:18:06.621 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:18:06.643704+0000 2024-03-29T11:18:06.621 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:06.621 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:06.621 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:06.621 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:18:06.621 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:18:06.621 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:06.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.643+0000 7fc9a359e640 1 Processor -- start 2024-03-29T11:18:06.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.643+0000 7fc9a359e640 1 -- start start 2024-03-29T11:18:06.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.643+0000 7fc9a359e640 10 monclient: init 2024-03-29T11:18:06.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.644+0000 7fc9a359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:06.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.644+0000 7fc9a359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:06.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.645+0000 7fc9a359e640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a4074650 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:06.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.645+0000 7fc9a359e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a406f3b0 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:06.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.645+0000 7fc9a359e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a4149ca0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:06.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.645+0000 7fc9a359e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a414a2f0 con 0x7fc9a4149ca0 2024-03-29T11:18:06.624 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.645+0000 7fc9a359e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a4072d60 con 0x7fc9a406f3b0 2024-03-29T11:18:06.624 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.645+0000 7fc9a359e640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a4072ef0 con 0x7fc9a4074650 2024-03-29T11:18:06.624 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.645+0000 7fc9a359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:06.624 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.646+0000 7fc9a359e640 10 monclient(hunting): authenticate will time out at 1298.842870s 2024-03-29T11:18:06.625 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a259c640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fc9a4074650 0x7fc9a4143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:38393/0 (socket says 172.21.15.157:38393) 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a259c640 1 -- 172.21.15.157:0/3134717688 learned_addr learned my addr 172.21.15.157:0/3134717688 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852431968 0 0) 0x7fc994001490 con 0x7fc9a4074650 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc990003520 con 0x7fc9a4074650 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242193985 0 0) 0x7fc9a4072d60 con 0x7fc9a406f3b0 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:18:06.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.647+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc994001490 con 0x7fc9a406f3b0 2024-03-29T11:18:06.628 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 693193808 0 0) 0x7fc990003520 con 0x7fc9a4074650 2024-03-29T11:18:06.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 >> v1:172.21.15.195:6789/0 conn(0x7fc9a406f3b0 legacy=0x7fc9a4146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:06.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 >> v1:172.21.15.157:6789/0 conn(0x7fc9a4149ca0 legacy=0x7fc9a4151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:06.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:18:06.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 1 monclient: found mon.noname-b 2024-03-29T11:18:06.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:06.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:18:06.630 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4143d10 con 0x7fc9a4074650 2024-03-29T11:18:06.630 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:18:06.630 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:18:06.630 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:06.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc994001a00 con 0x7fc9a4074650 2024-03-29T11:18:06.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:06.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:06.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a1d9b640 10 monclient: dump: 2024-03-29T11:18:06.631 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:06.631 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:06.631 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:06.631 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:06.632 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:06.632 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:06.632 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:06.632 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:06.632 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:06.632 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:06.632 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a359e640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:18:06.633 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.649+0000 7fc9a359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:06.633 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.650+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc994001c20 con 0x7fc9a4074650 2024-03-29T11:18:06.633 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.650+0000 7fc9a1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:06.633 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.650+0000 7fc9a1d9b640 1 -- 172.21.15.157:0/3134717688 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc994001fb0 con 0x7fc9a4074650 2024-03-29T11:18:06.633 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.650+0000 7fc9a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:06.633 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.650+0000 7fc9a1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:06.633 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.650+0000 7fc9a1d9b640 10 monclient: dump: 2024-03-29T11:18:06.633 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:06.634 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:06.634 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:06.634 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:06.634 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:06.634 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:06.634 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:06.635 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:06.635 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:06.635 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:06.635 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.650+0000 7fc9a359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:06.635 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.650+0000 7fc9a359e640 4 set_mon_vals no callback set 2024-03-29T11:18:06.635 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.651+0000 7fc9a359e640 10 monclient: shutdown 2024-03-29T11:18:06.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.651+0000 7fc9a359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:06.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.651+0000 7fc9a359e640 1 -- 172.21.15.157:0/3134717688 >> v1:172.21.15.157:6790/0 conn(0x7fc9a4074650 legacy=0x7fc9a4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:06.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.651+0000 7fc9a359e640 1 -- 172.21.15.157:0/3134717688 shutdown_connections 2024-03-29T11:18:06.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.651+0000 7fc9a359e640 1 -- 172.21.15.157:0/3134717688 >> 172.21.15.157:0/3134717688 conn(0x7fc9a40b4200 msgr2=0x7fc9a40b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:06.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.651+0000 7fc9a359e640 1 -- 172.21.15.157:0/3134717688 shutdown_connections 2024-03-29T11:18:06.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.651+0000 7fc9a359e640 1 -- 172.21.15.157:0/3134717688 wait complete. 2024-03-29T11:18:06.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.652+0000 7fc9a359e640 10 monclient: build_initial_monmap 2024-03-29T11:18:06.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.652+0000 7fc9a359e640 10 monclient: monmap: 2024-03-29T11:18:06.648 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:06.648 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:06.648 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:18:06.649 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:18:06.649 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:06.649 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:06.649 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:06.649 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:18:06.649 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:18:06.649 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:06.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.653+0000 7fc9a359e640 1 librados: starting msgr at 2024-03-29T11:18:06.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.653+0000 7fc9a359e640 1 librados: starting objecter 2024-03-29T11:18:06.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.653+0000 7fc9a359e640 1 Processor -- start 2024-03-29T11:18:06.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.653+0000 7fc9a359e640 1 -- start start 2024-03-29T11:18:06.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.653+0000 7fc9a359e640 1 librados: setting wanted keys 2024-03-29T11:18:06.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.653+0000 7fc9a359e640 1 librados: calling monclient init 2024-03-29T11:18:06.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.653+0000 7fc9a359e640 10 monclient: init 2024-03-29T11:18:06.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.654+0000 7fc9a359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:06.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.654+0000 7fc9a359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:06.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.655+0000 7fc9a359e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a406f3b0 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:06.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.655+0000 7fc9a359e640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a4074650 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:06.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.655+0000 7fc9a359e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a4149ca0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:06.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.655+0000 7fc9a359e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a41decb0 con 0x7fc9a4149ca0 2024-03-29T11:18:06.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.655+0000 7fc9a359e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a40acb20 con 0x7fc9a4074650 2024-03-29T11:18:06.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.655+0000 7fc9a359e640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a41def00 con 0x7fc9a406f3b0 2024-03-29T11:18:06.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.655+0000 7fc9a359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:06.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.655+0000 7fc9a359e640 10 monclient(hunting): authenticate will time out at 1298.852400s 2024-03-29T11:18:06.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.656+0000 7fc99bfff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc9a4074650 0x7fc9a40a94e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:43460/0 (socket says 172.21.15.157:43460) 2024-03-29T11:18:06.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.656+0000 7fc99bfff640 1 -- 172.21.15.157:0/210620287 learned_addr learned my addr 172.21.15.157:0/210620287 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:06.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.657+0000 7fc99affd640 10 client.?.objecter ms_handle_connect 0x7fc9a4074650 2024-03-29T11:18:06.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.657+0000 7fc99affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:06.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.659+0000 7fc99affd640 10 client.?.objecter ms_handle_connect 0x7fc9a406f3b0 2024-03-29T11:18:06.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.659+0000 7fc99affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:06.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.659+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710678483 0 0) 0x7fc9a40acb20 con 0x7fc9a4074650 2024-03-29T11:18:06.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.659+0000 7fc99affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:06.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.659+0000 7fc99affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:06.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.659+0000 7fc99affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:06.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.659+0000 7fc99affd640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:18:06.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc984004eb0 con 0x7fc9a4074650 2024-03-29T11:18:06.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645736726 0 0) 0x7fc9a41def00 con 0x7fc9a406f3b0 2024-03-29T11:18:06.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:06.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:06.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:06.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:18:06.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a40acb20 con 0x7fc9a406f3b0 2024-03-29T11:18:06.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 10 client.?.objecter ms_handle_connect 0x7fc9a4149ca0 2024-03-29T11:18:06.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:06.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3828270023 0 0) 0x7fc984004eb0 con 0x7fc9a4074650 2024-03-29T11:18:06.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9a41def00 con 0x7fc9a4074650 2024-03-29T11:18:06.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122725041 0 0) 0x7fc9a41decb0 con 0x7fc9a4149ca0 2024-03-29T11:18:06.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:06.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:06.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.660+0000 7fc99affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:06.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:18:06.655 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc984004eb0 con 0x7fc9a4149ca0 2024-03-29T11:18:06.655 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2865715664 0 0) 0x7fc9a40acb20 con 0x7fc9a406f3b0 2024-03-29T11:18:06.655 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9a41decb0 con 0x7fc9a406f3b0 2024-03-29T11:18:06.655 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc98c001a70 con 0x7fc9a4074650 2024-03-29T11:18:06.655 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:06.655 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:06.655 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient(hunting): dump: 2024-03-29T11:18:06.655 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:06.656 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:06.656 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:06.656 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:06.656 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:06.656 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:06.656 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:06.656 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:06.656 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc994004b60 con 0x7fc9a406f3b0 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient(hunting): dump: 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:06.657 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1391882816 0 0) 0x7fc9a41decb0 con 0x7fc9a406f3b0 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 >> v1:172.21.15.195:6789/0 conn(0x7fc9a4074650 legacy=0x7fc9a40a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 >> v1:172.21.15.157:6789/0 conn(0x7fc9a4149ca0 legacy=0x7fc9a413f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 1 monclient: found mon.c 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a41df420 con 0x7fc9a406f3b0 2024-03-29T11:18:06.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient: _finish_auth 0 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.661+0000 7fc99affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.662+0000 7fc9a359e640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.662+0000 7fc9a359e640 10 monclient: _renew_subs 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.662+0000 7fc9a359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.662+0000 7fc9a359e640 1 -- 172.21.15.157:0/210620287 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a40a9bf0 con 0x7fc9a406f3b0 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.662+0000 7fc9a359e640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.662+0000 7fc9a359e640 10 monclient: _renew_subs 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.662+0000 7fc9a359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.662+0000 7fc9a359e640 1 -- 172.21.15.157:0/210620287 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a4148550 con 0x7fc9a406f3b0 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.662+0000 7fc9a359e640 1 librados: init done 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc998ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc998ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc998ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:18:06.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc998ff9640 1 -- 172.21.15.157:0/210620287 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a41df090 con 0x7fc9a406f3b0 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc994001740 con 0x7fc9a406f3b0 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc99affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc994005110 con 0x7fc9a406f3b0 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc99affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc99affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc99affd640 10 monclient: dump: 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:06.660 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:06.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.664+0000 7fc9a159a640 4 set_mon_vals no callback set 2024-03-29T11:18:06.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.673+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fc99401b0e0 con 0x7fc9a406f3b0 2024-03-29T11:18:06.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.674+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fc994054bf0 con 0x7fc9a406f3b0 2024-03-29T11:18:06.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.674+0000 7fc99affd640 10 client.4487.objecter ms_dispatch 0x7fc9a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:06.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.674+0000 7fc99affd640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:06.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.674+0000 7fc99affd640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:06.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.674+0000 7fc99affd640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T11:18:06.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.674+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc994050e90 con 0x7fc9a406f3b0 2024-03-29T11:18:06.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.674+0000 7fc99affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:06.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.674+0000 7fc99affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:06.662 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.674+0000 7fc99bfff640 10 monclient: get_auth_request con 0x7fc98406dff0 auth_method 0 2024-03-29T11:18:06.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:06.675+0000 7fc99affd640 10 client.4487.objecter ms_handle_connect 0x7fc98406dff0 2024-03-29T11:18:06.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.001+0000 7fc998ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:06.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.001+0000 7fc998ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:06.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.001+0000 7fc998ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:18:06.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.001+0000 7fc998ff9640 1 -- 172.21.15.157:0/210620287 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9a4067fe0 con 0x7fc9a406f3b0 2024-03-29T11:18:06.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.003+0000 7fc99affd640 1 -- 172.21.15.157:0/210620287 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fc994050e90 con 0x7fc9a406f3b0 2024-03-29T11:18:06.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.003+0000 7fc99affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:06.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.003+0000 7fc99affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:06.981 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:18:06.981 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:06.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.007+0000 7fc998ff9640 10 librados: watch_flush enter 2024-03-29T11:18:06.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.007+0000 7fc998ff9640 10 librados: watch_flush exit 2024-03-29T11:18:06.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.007+0000 7fc998ff9640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T11:18:06.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.007+0000 7fc998ff9640 10 client.4487.objecter successfully canceled tick 2024-03-29T11:18:06.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.008+0000 7fc998ff9640 1 -- 172.21.15.157:0/210620287 >> v1:172.21.15.157:6816/1747058247 conn(0x7fc98406dff0 legacy=0x7fc9840704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:06.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.008+0000 7fc998ff9640 10 monclient: shutdown 2024-03-29T11:18:06.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.008+0000 7fc998ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:06.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.008+0000 7fc998ff9640 1 -- 172.21.15.157:0/210620287 >> v1:172.21.15.157:6790/0 conn(0x7fc9a406f3b0 legacy=0x7fc9a41de5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:06.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.008+0000 7fc998ff9640 1 -- 172.21.15.157:0/210620287 shutdown_connections 2024-03-29T11:18:06.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.008+0000 7fc998ff9640 1 -- 172.21.15.157:0/210620287 >> 172.21.15.157:0/210620287 conn(0x7fc9a40b4200 msgr2=0x7fc9a40b45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:06.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.010+0000 7fc998ff9640 1 -- 172.21.15.157:0/210620287 shutdown_connections 2024-03-29T11:18:06.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.010+0000 7fc998ff9640 1 -- 172.21.15.157:0/210620287 wait complete. 2024-03-29T11:18:06.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:07.010+0000 7fc998ff9640 1 librados: shutdown 2024-03-29T11:18:07.005 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:13.812 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:13.958 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.980+0000 7f163099d640 10 monclient: get_monmap_and_config 2024-03-29T11:18:13.958 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.980+0000 7f163099d640 10 monclient: build_initial_monmap 2024-03-29T11:18:13.958 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.980+0000 7f163099d640 10 monclient: monmap: 2024-03-29T11:18:13.958 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:13.958 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:13.958 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:18:13.981849+0000 2024-03-29T11:18:13.958 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:18:13.981849+0000 2024-03-29T11:18:13.958 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:13.959 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:13.959 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:13.959 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:18:13.959 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:18:13.959 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:13.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.981+0000 7f163099d640 1 Processor -- start 2024-03-29T11:18:13.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.981+0000 7f163099d640 1 -- start start 2024-03-29T11:18:13.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.981+0000 7f163099d640 10 monclient: init 2024-03-29T11:18:13.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.982+0000 7f163099d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:13.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.982+0000 7f163099d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:13.961 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.982+0000 7f163099d640 10 monclient(hunting): picked mon.noname-c con 0x7f162c0b0770 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.982+0000 7f163099d640 10 monclient(hunting): picked mon.noname-a con 0x7f162c14a700 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.982+0000 7f163099d640 10 monclient(hunting): picked mon.noname-b con 0x7f162c0ac5a0 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.982+0000 7f163099d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c0aba70 con 0x7f162c14a700 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.982+0000 7f163099d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c0aecb0 con 0x7f162c0b0770 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.982+0000 7f163099d640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c0af150 con 0x7f162c0ac5a0 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.982+0000 7f163099d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.982+0000 7f163099d640 10 monclient(hunting): authenticate will time out at 1306.179806s 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162ad9d640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f162c0ac5a0 0x7f162c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:38248/0 (socket says 172.21.15.157:38248) 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162ad9d640 1 -- 172.21.15.157:0/91635877 learned_addr learned my addr 172.21.15.157:0/91635877 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 1 -- 172.21.15.157:0/91635877 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709293038 0 0) 0x7f162c0af150 con 0x7f162c0ac5a0 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 10 monclient(hunting): _negotiate 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:13.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 1 -- 172.21.15.157:0/91635877 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1610003520 con 0x7f162c0ac5a0 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 1 -- 172.21.15.157:0/91635877 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1892841504 0 0) 0x7f1610003520 con 0x7f162c0ac5a0 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 1 -- 172.21.15.157:0/91635877 >> v1:172.21.15.195:6789/0 conn(0x7f162c0b0770 legacy=0x7f162c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 1 -- 172.21.15.157:0/91635877 >> v1:172.21.15.157:6789/0 conn(0x7f162c14a700 legacy=0x7f162c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 10 monclient: _finish_hunting 0 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 1 monclient: found mon.noname-b 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 1 -- 172.21.15.157:0/91635877 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c0ac9d0 con 0x7f162c0ac5a0 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 10 monclient: _finish_auth 0 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 10 monclient: _check_auth_tickets 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:13.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f162959a640 1 -- 172.21.15.157:0/91635877 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1624001a10 con 0x7f162c0ac5a0 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.983+0000 7f163099d640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f163099d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f162959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f162959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f162959a640 10 monclient: dump: 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:13.964 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f163099d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f162959a640 1 -- 172.21.15.157:0/91635877 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1624001b70 con 0x7f162c0ac5a0 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f162959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f162959a640 1 -- 172.21.15.157:0/91635877 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1624001f80 con 0x7f162c0ac5a0 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f162959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f162959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f162959a640 10 monclient: dump: 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:13.965 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:13.966 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:13.966 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:13.966 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:13.966 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f163099d640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:13.966 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.984+0000 7f163099d640 4 set_mon_vals no callback set 2024-03-29T11:18:13.974 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.985+0000 7f163099d640 10 monclient: shutdown 2024-03-29T11:18:13.974 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.985+0000 7f163099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:13.974 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.985+0000 7f163099d640 1 -- 172.21.15.157:0/91635877 >> v1:172.21.15.157:6790/0 conn(0x7f162c0ac5a0 legacy=0x7f162c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.974 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.985+0000 7f163099d640 1 -- 172.21.15.157:0/91635877 shutdown_connections 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.985+0000 7f163099d640 1 -- 172.21.15.157:0/91635877 >> 172.21.15.157:0/91635877 conn(0x7f162c06b660 msgr2=0x7f162c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.985+0000 7f163099d640 1 -- 172.21.15.157:0/91635877 shutdown_connections 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.985+0000 7f163099d640 1 -- 172.21.15.157:0/91635877 wait complete. 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.986+0000 7f163099d640 10 monclient: build_initial_monmap 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.986+0000 7f163099d640 10 monclient: monmap: 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:18:13.975 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:13.976 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.987+0000 7f163099d640 1 librados: starting msgr at 2024-03-29T11:18:13.976 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.987+0000 7f163099d640 1 librados: starting objecter 2024-03-29T11:18:13.977 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.987+0000 7f163099d640 1 Processor -- start 2024-03-29T11:18:13.977 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.987+0000 7f163099d640 1 -- start start 2024-03-29T11:18:13.977 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.987+0000 7f163099d640 1 librados: setting wanted keys 2024-03-29T11:18:13.977 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.987+0000 7f163099d640 1 librados: calling monclient init 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.987+0000 7f163099d640 10 monclient: init 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f163099d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f163099d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f163099d640 10 monclient(hunting): picked mon.noname-a con 0x7f162c0ac5a0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f163099d640 10 monclient(hunting): picked mon.noname-c con 0x7f162c0b0770 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f163099d640 10 monclient(hunting): picked mon.noname-b con 0x7f162c14a700 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f163099d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c1281c0 con 0x7f162c0ac5a0 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f163099d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c1284a0 con 0x7f162c14a700 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f162ad9d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f162c14a700 0x7f162c1dbf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:55326/0 (socket says 172.21.15.157:55326) 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f162ad9d640 1 -- 172.21.15.157:0/1309114865 learned_addr learned my addr 172.21.15.157:0/1309114865 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c127270 con 0x7f162c0b0770 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f163099d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:13.978 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.988+0000 7f163099d640 10 monclient(hunting): authenticate will time out at 1306.185594s 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 client.?.objecter ms_handle_connect 0x7f162c14a700 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 client.?.objecter ms_handle_connect 0x7f162c0ac5a0 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 client.?.objecter ms_handle_connect 0x7f162c0b0770 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600379798 0 0) 0x7f162c1281c0 con 0x7f162c0ac5a0 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1614004eb0 con 0x7f162c0ac5a0 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197274697 0 0) 0x7f162c1284a0 con 0x7f162c14a700 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:13.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162c1281c0 con 0x7f162c14a700 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263674063 0 0) 0x7f162c127270 con 0x7f162c0b0770 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.989+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162c1284a0 con 0x7f162c0b0770 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 235498784 0 0) 0x7f1614004eb0 con 0x7f162c0ac5a0 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f162c127270 con 0x7f162c0ac5a0 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f161c0018b0 con 0x7f162c0ac5a0 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:13.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 10 monclient(hunting): dump: 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 359538220 0 0) 0x7f162c1281c0 con 0x7f162c14a700 2024-03-29T11:18:13.981 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1614004eb0 con 0x7f162c14a700 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1624001ec0 con 0x7f162c14a700 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.990+0000 7f1622ffd640 10 monclient(hunting): dump: 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:13.982 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2634175748 0 0) 0x7f162c1284a0 con 0x7f162c0b0770 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f162c1281c0 con 0x7f162c0b0770 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f1618001780 con 0x7f162c0b0770 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 10 monclient(hunting): dump: 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:13.983 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:13.984 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:13.984 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:13.984 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:13.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3314849959 0 0) 0x7f162c127270 con 0x7f162c0ac5a0 2024-03-29T11:18:13.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 >> v1:172.21.15.157:6790/0 conn(0x7f162c0b0770 legacy=0x7f162c1db810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 >> v1:172.21.15.195:6789/0 conn(0x7f162c14a700 legacy=0x7f162c1dbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:13.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:13.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 1 monclient: found mon.a 2024-03-29T11:18:13.984 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c1d9110 con 0x7f162c0ac5a0 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 10 monclient: _finish_auth 0 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f163099d640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f163099d640 10 monclient: _renew_subs 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f163099d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162c126e60 con 0x7f162c0ac5a0 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f163099d640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f163099d640 10 monclient: _renew_subs 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f163099d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f162c1459a0 con 0x7f162c0ac5a0 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f163099d640 1 librados: init done 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f161c0035f0 con 0x7f162c0ac5a0 2024-03-29T11:18:13.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.991+0000 7f1622ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.992+0000 7f1623fff640 4 set_mon_vals no callback set 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.992+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f161c003900 con 0x7f162c0ac5a0 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.992+0000 7f1622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.992+0000 7f1622ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.992+0000 7f1622ffd640 10 monclient: dump: 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:13.986 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.993+0000 7f163099d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.993+0000 7f163099d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.993+0000 7f163099d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.993+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1608005080 con 0x7f162c0ac5a0 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.994+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f161c0198d0 con 0x7f162c0ac5a0 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.995+0000 7f1629d9b640 10 monclient: get_auth_request con 0x7f161406e0c0 auth_method 0 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.995+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f161c083840 con 0x7f162c0ac5a0 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.995+0000 7f1622ffd640 10 client.4473.objecter ms_dispatch 0x7f162c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.995+0000 7f1622ffd640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.995+0000 7f1622ffd640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.995+0000 7f1622ffd640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T11:18:13.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:13.995+0000 7f1622ffd640 10 client.4473.objecter ms_handle_connect 0x7f161406e0c0 2024-03-29T11:18:13.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.006+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f161c04ef50 con 0x7f162c0ac5a0 2024-03-29T11:18:13.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.006+0000 7f1622ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:13.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.007+0000 7f1622ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:14.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.385+0000 7f163099d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:14.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.385+0000 7f163099d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:14.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.385+0000 7f163099d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:18:14.363 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.385+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1608005330 con 0x7f162c0ac5a0 2024-03-29T11:18:14.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.387+0000 7f1622ffd640 1 -- 172.21.15.157:0/1309114865 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f161c051050 con 0x7f162c0ac5a0 2024-03-29T11:18:14.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.387+0000 7f1622ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:14.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.387+0000 7f1622ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:14.364 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:18:14.364 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:14.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.390+0000 7f163099d640 10 librados: watch_flush enter 2024-03-29T11:18:14.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.390+0000 7f163099d640 10 librados: watch_flush exit 2024-03-29T11:18:14.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.390+0000 7f163099d640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T11:18:14.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.390+0000 7f163099d640 10 client.4473.objecter successfully canceled tick 2024-03-29T11:18:14.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.390+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 >> v1:172.21.15.157:6816/1747058247 conn(0x7f161406e0c0 legacy=0x7f1614070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:14.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.391+0000 7f163099d640 10 monclient: shutdown 2024-03-29T11:18:14.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.391+0000 7f163099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:14.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.391+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 >> v1:172.21.15.157:6789/0 conn(0x7f162c0ac5a0 legacy=0x7f162c1450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:14.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.391+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 shutdown_connections 2024-03-29T11:18:14.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.391+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 >> 172.21.15.157:0/1309114865 conn(0x7f162c06b660 msgr2=0x7f162c1494a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:14.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.391+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 shutdown_connections 2024-03-29T11:18:14.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.391+0000 7f163099d640 1 -- 172.21.15.157:0/1309114865 wait complete. 2024-03-29T11:18:14.369 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:14.392+0000 7f163099d640 1 librados: shutdown 2024-03-29T11:18:14.388 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:21.194 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:21.328 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.351+0000 7f52d6e50640 10 monclient: get_monmap_and_config 2024-03-29T11:18:21.328 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.351+0000 7f52d6e50640 10 monclient: build_initial_monmap 2024-03-29T11:18:21.328 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.352+0000 7f52d6e50640 10 monclient: monmap: 2024-03-29T11:18:21.328 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:21.328 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:21.328 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:18:21.352088+0000 2024-03-29T11:18:21.328 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:18:21.352088+0000 2024-03-29T11:18:21.328 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:21.328 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:21.328 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:21.329 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:18:21.329 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:18:21.329 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:21.329 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.353+0000 7f52d6e50640 1 Processor -- start 2024-03-29T11:18:21.329 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.353+0000 7f52d6e50640 1 -- start start 2024-03-29T11:18:21.329 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.353+0000 7f52d6e50640 10 monclient: init 2024-03-29T11:18:21.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.353+0000 7f52d6e50640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:21.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.353+0000 7f52d6e50640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:21.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.354+0000 7f52d6e50640 10 monclient(hunting): picked mon.noname-a con 0x7f52d01457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:21.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.354+0000 7f52d6e50640 10 monclient(hunting): picked mon.noname-c con 0x7f52d013f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:21.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.354+0000 7f52d6e50640 10 monclient(hunting): picked mon.noname-b con 0x7f52d00a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:21.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.354+0000 7f52d6e50640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d00aaef0 con 0x7f52d01457f0 2024-03-29T11:18:21.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.354+0000 7f52d6e50640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d0145bd0 con 0x7f52d013f650 2024-03-29T11:18:21.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.354+0000 7f52d6e50640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d0145eb0 con 0x7f52d00a9d70 2024-03-29T11:18:21.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.354+0000 7f52d6e50640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:21.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.354+0000 7f52d6e50640 10 monclient(hunting): authenticate will time out at 1313.549957s 2024-03-29T11:18:21.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.354+0000 7f52d53ed640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f52d00a9d70 0x7f52d014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:56604/0 (socket says 172.21.15.157:56604) 2024-03-29T11:18:21.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.354+0000 7f52d53ed640 1 -- 172.21.15.157:0/614310327 learned_addr learned my addr 172.21.15.157:0/614310327 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:21.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.355+0000 7f52cf7fe640 1 -- 172.21.15.157:0/614310327 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181254737 0 0) 0x7f52d0145eb0 con 0x7f52d00a9d70 2024-03-29T11:18:21.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.355+0000 7f52cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:21.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.355+0000 7f52cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:21.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.355+0000 7f52cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:21.332 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.355+0000 7f52cf7fe640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:18:21.332 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.355+0000 7f52cf7fe640 1 -- 172.21.15.157:0/614310327 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c0003520 con 0x7f52d00a9d70 2024-03-29T11:18:21.332 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.355+0000 7f52cf7fe640 1 -- 172.21.15.157:0/614310327 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 373345539 0 0) 0x7f52c0003520 con 0x7f52d00a9d70 2024-03-29T11:18:21.332 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.356+0000 7f52cf7fe640 1 -- 172.21.15.157:0/614310327 >> v1:172.21.15.195:6789/0 conn(0x7f52d013f650 legacy=0x7f52d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:21.332 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.356+0000 7f52cf7fe640 1 -- 172.21.15.157:0/614310327 >> v1:172.21.15.157:6789/0 conn(0x7f52d01457f0 legacy=0x7f52d013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:21.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:21.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 1 monclient: found mon.noname-b 2024-03-29T11:18:21.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:21.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:18:21.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 1 -- 172.21.15.157:0/614310327 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d0146340 con 0x7f52d00a9d70 2024-03-29T11:18:21.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:21.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:21.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:21.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 1 -- 172.21.15.157:0/614310327 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f52c8001a10 con 0x7f52d00a9d70 2024-03-29T11:18:21.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: dump: 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52d6e50640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:18:21.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52d6e50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:21.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 1 -- 172.21.15.157:0/614310327 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52c8001c70 con 0x7f52d00a9d70 2024-03-29T11:18:21.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:21.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 1 -- 172.21.15.157:0/614310327 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f52c80020c0 con 0x7f52d00a9d70 2024-03-29T11:18:21.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52d6e50640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:21.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52d6e50640 4 set_mon_vals no callback set 2024-03-29T11:18:21.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:21.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:21.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.358+0000 7f52cf7fe640 10 monclient: dump: 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.359+0000 7f52d6e50640 10 monclient: shutdown 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.359+0000 7f52d6e50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.359+0000 7f52d6e50640 1 -- 172.21.15.157:0/614310327 >> v1:172.21.15.157:6790/0 conn(0x7f52d00a9d70 legacy=0x7f52d014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:21.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.359+0000 7f52d6e50640 1 -- 172.21.15.157:0/614310327 shutdown_connections 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.359+0000 7f52d6e50640 1 -- 172.21.15.157:0/614310327 >> 172.21.15.157:0/614310327 conn(0x7f52d0066a00 msgr2=0x7f52d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.359+0000 7f52d6e50640 1 -- 172.21.15.157:0/614310327 shutdown_connections 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.359+0000 7f52d6e50640 1 -- 172.21.15.157:0/614310327 wait complete. 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.360+0000 7f52d6e50640 10 monclient: build_initial_monmap 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.360+0000 7f52d6e50640 10 monclient: monmap: 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:18:21.339 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:21.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.361+0000 7f52d6e50640 1 librados: starting msgr at 2024-03-29T11:18:21.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.361+0000 7f52d6e50640 1 librados: starting objecter 2024-03-29T11:18:21.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.361+0000 7f52d6e50640 1 Processor -- start 2024-03-29T11:18:21.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.361+0000 7f52d6e50640 1 -- start start 2024-03-29T11:18:21.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.361+0000 7f52d6e50640 1 librados: setting wanted keys 2024-03-29T11:18:21.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.361+0000 7f52d6e50640 1 librados: calling monclient init 2024-03-29T11:18:21.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.361+0000 7f52d6e50640 10 monclient: init 2024-03-29T11:18:21.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d6e50640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d6e50640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d6e50640 10 monclient(hunting): picked mon.noname-a con 0x7f52d00a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d6e50640 10 monclient(hunting): picked mon.noname-b con 0x7f52d013f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d6e50640 10 monclient(hunting): picked mon.noname-c con 0x7f52d01457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d6e50640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d0142e60 con 0x7f52d00a9d70 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d6e50640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d0141af0 con 0x7f52d013f650 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d6e50640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d0140b00 con 0x7f52d01457f0 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d6e50640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d6e50640 10 monclient(hunting): authenticate will time out at 1313.558067s 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d53ed640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f52d01457f0 0x7f52d01e16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:58926/0 (socket says 172.21.15.157:58926) 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52d53ed640 1 -- 172.21.15.157:0/106001030 learned_addr learned my addr 172.21.15.157:0/106001030 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 client.?.objecter ms_handle_connect 0x7f52d01457f0 2024-03-29T11:18:21.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 client.?.objecter ms_handle_connect 0x7f52d013f650 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875746410 0 0) 0x7f52d0140b00 con 0x7f52d01457f0 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b4004eb0 con 0x7f52d01457f0 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568618741 0 0) 0x7f52d0141af0 con 0x7f52d013f650 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:21.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.362+0000 7f52cd7fa640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d0140b00 con 0x7f52d013f650 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4200670265 0 0) 0x7f52b4004eb0 con 0x7f52d01457f0 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52d0141af0 con 0x7f52d01457f0 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f52c8001830 con 0x7f52d01457f0 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 10 monclient(hunting): dump: 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:21.350 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1917410722 0 0) 0x7f52d0140b00 con 0x7f52d013f650 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52b4004eb0 con 0x7f52d013f650 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f52c4001510 con 0x7f52d013f650 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 10 monclient(hunting): dump: 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:21.351 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3861468565 0 0) 0x7f52d0141af0 con 0x7f52d01457f0 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 >> v1:172.21.15.195:6789/0 conn(0x7f52d013f650 legacy=0x7f52d0140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.363+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 >> v1:172.21.15.157:6789/0 conn(0x7f52d00a9d70 legacy=0x7f52d013b3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52cd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52cd7fa640 1 monclient: found mon.c 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:21.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52cd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d0140820 con 0x7f52d01457f0 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52cd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52cd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52d6e50640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52d6e50640 10 monclient: _renew_subs 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52d6e50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.365+0000 7f52d6e50640 1 -- 172.21.15.157:0/106001030 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d01427b0 con 0x7f52d01457f0 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52d6e50640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52d6e50640 10 monclient: _renew_subs 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52d6e50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52d6e50640 1 -- 172.21.15.157:0/106001030 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d006de90 con 0x7f52d01457f0 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52d6e50640 1 librados: init done 2024-03-29T11:18:21.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52c8001990 con 0x7f52d01457f0 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52ceffd640 4 set_mon_vals no callback set 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f52c8005190 con 0x7f52d01457f0 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.366+0000 7f52cd7fa640 10 monclient: dump: 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:21.354 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.367+0000 7f52d6e50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.367+0000 7f52d6e50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.367+0000 7f52d6e50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.367+0000 7f52d6e50640 1 -- 172.21.15.157:0/106001030 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b0005080 con 0x7f52d01457f0 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.368+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f52c801b160 con 0x7f52d01457f0 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.369+0000 7f52d4bec640 10 monclient: get_auth_request con 0x7f52b407f090 auth_method 0 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.369+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f52c8084f80 con 0x7f52d01457f0 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.369+0000 7f52cd7fa640 10 client.4499.objecter ms_dispatch 0x7f52d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.369+0000 7f52cd7fa640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.369+0000 7f52cd7fa640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.369+0000 7f52cd7fa640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:18:21.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.369+0000 7f52cd7fa640 10 client.4499.objecter ms_handle_connect 0x7f52b407f090 2024-03-29T11:18:21.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.376+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52c8050860 con 0x7f52d01457f0 2024-03-29T11:18:21.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.376+0000 7f52cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:21.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.376+0000 7f52cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:21.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.692+0000 7f52d6e50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:21.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.692+0000 7f52d6e50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:21.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.692+0000 7f52d6e50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:18:21.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.692+0000 7f52d6e50640 1 -- 172.21.15.157:0/106001030 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52b0005330 con 0x7f52d01457f0 2024-03-29T11:18:21.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.693+0000 7f52cd7fa640 1 -- 172.21.15.157:0/106001030 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f52c8052790 con 0x7f52d01457f0 2024-03-29T11:18:21.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.693+0000 7f52cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:21.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.693+0000 7f52cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:21.670 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:18:21.670 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:21.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 10 librados: watch_flush enter 2024-03-29T11:18:21.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 10 librados: watch_flush exit 2024-03-29T11:18:21.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:18:21.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:18:21.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 1 -- 172.21.15.157:0/106001030 >> v1:172.21.15.157:6816/1747058247 conn(0x7f52b407f090 legacy=0x7f52b4081550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:21.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 10 monclient: shutdown 2024-03-29T11:18:21.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:21.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 1 -- 172.21.15.157:0/106001030 >> v1:172.21.15.157:6790/0 conn(0x7f52d01457f0 legacy=0x7f52d01e16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:21.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 1 -- 172.21.15.157:0/106001030 shutdown_connections 2024-03-29T11:18:21.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 1 -- 172.21.15.157:0/106001030 >> 172.21.15.157:0/106001030 conn(0x7f52d0066a00 msgr2=0x7f52d00b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:21.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 1 -- 172.21.15.157:0/106001030 shutdown_connections 2024-03-29T11:18:21.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 1 -- 172.21.15.157:0/106001030 wait complete. 2024-03-29T11:18:21.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:21.697+0000 7f52d6e50640 1 librados: shutdown 2024-03-29T11:18:21.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:28.497 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:28.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.670+0000 7febed86d640 10 monclient: get_monmap_and_config 2024-03-29T11:18:28.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.671+0000 7febed86d640 10 monclient: build_initial_monmap 2024-03-29T11:18:28.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.671+0000 7febed86d640 10 monclient: monmap: 2024-03-29T11:18:28.649 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:28.649 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:28.649 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:18:28.672120+0000 2024-03-29T11:18:28.650 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:18:28.672120+0000 2024-03-29T11:18:28.650 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:28.650 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:28.650 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:28.650 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:18:28.650 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:18:28.650 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:28.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.673+0000 7febed86d640 1 Processor -- start 2024-03-29T11:18:28.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.673+0000 7febed86d640 1 -- start start 2024-03-29T11:18:28.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.673+0000 7febed86d640 10 monclient: init 2024-03-29T11:18:28.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febed86d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:28.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febed86d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:28.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febed86d640 10 monclient(hunting): picked mon.noname-a con 0x7febe80a9ac0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:28.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febed86d640 10 monclient(hunting): picked mon.noname-b con 0x7febe813ee90 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:28.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febed86d640 10 monclient(hunting): picked mon.noname-c con 0x7febe80af900 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:28.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febed86d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe80afd10 con 0x7febe80a9ac0 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febed86d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe80af6f0 con 0x7febe80af900 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febed86d640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe80aac80 con 0x7febe813ee90 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febed86d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febed86d640 10 monclient(hunting): authenticate will time out at 1320.871005s 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febe67fc640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7febe813ee90 0x7febe813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:54486/0 (socket says 172.21.15.157:54486) 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febe67fc640 1 -- 172.21.15.157:0/3176160565 learned_addr learned my addr 172.21.15.157:0/3176160565 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febe5ffb640 1 -- 172.21.15.157:0/3176160565 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741695026 0 0) 0x7febe80aac80 con 0x7febe813ee90 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febe5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febe5ffb640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.674+0000 7febe5ffb640 1 -- 172.21.15.157:0/3176160565 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd0003520 con 0x7febe813ee90 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.675+0000 7febe5ffb640 1 -- 172.21.15.157:0/3176160565 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566997063 0 0) 0x7febd0003520 con 0x7febe813ee90 2024-03-29T11:18:28.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.675+0000 7febe5ffb640 1 -- 172.21.15.157:0/3176160565 >> v1:172.21.15.195:6789/0 conn(0x7febe80af900 legacy=0x7febe8149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.677+0000 7febe5ffb640 1 -- 172.21.15.157:0/3176160565 >> v1:172.21.15.157:6789/0 conn(0x7febe80a9ac0 legacy=0x7febe80adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.677+0000 7febe5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.677+0000 7febe5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.677+0000 7febe5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.677+0000 7febe5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.677+0000 7febe5ffb640 1 -- 172.21.15.157:0/3176160565 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe80aaf60 con 0x7febe813ee90 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 1 -- 172.21.15.157:0/3176160565 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7febe0001a10 con 0x7febe813ee90 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:28.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 10 monclient: dump: 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 1 -- 172.21.15.157:0/3176160565 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febe0001ca0 con 0x7febe813ee90 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 1 -- 172.21.15.157:0/3176160565 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7febe0002030 con 0x7febe813ee90 2024-03-29T11:18:28.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.678+0000 7febe5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.679+0000 7febe5ffb640 10 monclient: dump: 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:28.658 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.679+0000 7febed86d640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.679+0000 7febed86d640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.679+0000 7febed86d640 4 set_mon_vals no callback set 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.680+0000 7febed86d640 10 monclient: shutdown 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.680+0000 7febed86d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.680+0000 7febed86d640 1 -- 172.21.15.157:0/3176160565 >> v1:172.21.15.157:6790/0 conn(0x7febe813ee90 legacy=0x7febe813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.680+0000 7febed86d640 1 -- 172.21.15.157:0/3176160565 shutdown_connections 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.680+0000 7febed86d640 1 -- 172.21.15.157:0/3176160565 >> 172.21.15.157:0/3176160565 conn(0x7febe80b35e0 msgr2=0x7febe80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.681+0000 7febed86d640 1 -- 172.21.15.157:0/3176160565 shutdown_connections 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.681+0000 7febed86d640 1 -- 172.21.15.157:0/3176160565 wait complete. 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.682+0000 7febed86d640 10 monclient: build_initial_monmap 2024-03-29T11:18:28.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.682+0000 7febed86d640 10 monclient: monmap: 2024-03-29T11:18:28.661 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:28.661 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:28.661 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:18:28.661 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:18:28.661 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:28.661 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:28.661 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:28.661 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:18:28.661 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:18:28.661 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:28.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.683+0000 7febed86d640 1 librados: starting msgr at 2024-03-29T11:18:28.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.683+0000 7febed86d640 1 librados: starting objecter 2024-03-29T11:18:28.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.683+0000 7febed86d640 1 Processor -- start 2024-03-29T11:18:28.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 1 -- start start 2024-03-29T11:18:28.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 1 librados: setting wanted keys 2024-03-29T11:18:28.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 1 librados: calling monclient init 2024-03-29T11:18:28.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 10 monclient: init 2024-03-29T11:18:28.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:28.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:28.663 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 10 monclient(hunting): picked mon.noname-a con 0x7febe80a9ac0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:28.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 10 monclient(hunting): picked mon.noname-b con 0x7febe80af900 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:28.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 10 monclient(hunting): picked mon.noname-c con 0x7febe813ee90 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:28.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe8141890 con 0x7febe80a9ac0 2024-03-29T11:18:28.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe8141b70 con 0x7febe80af900 2024-03-29T11:18:28.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe81421a0 con 0x7febe813ee90 2024-03-29T11:18:28.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:28.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.684+0000 7febed86d640 10 monclient(hunting): authenticate will time out at 1320.881762s 2024-03-29T11:18:28.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.687+0000 7febe77fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7febe813ee90 0x7febe81de740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:38576/0 (socket says 172.21.15.157:38576) 2024-03-29T11:18:28.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.687+0000 7febe77fe640 1 -- 172.21.15.157:0/1655165562 learned_addr learned my addr 172.21.15.157:0/1655165562 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:28.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 10 client.?.objecter ms_handle_connect 0x7febe80a9ac0 2024-03-29T11:18:28.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:28.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 10 client.?.objecter ms_handle_connect 0x7febe813ee90 2024-03-29T11:18:28.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:28.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817083070 0 0) 0x7febe8141890 con 0x7febe80a9ac0 2024-03-29T11:18:28.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:28.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:28.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:28.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd4004eb0 con 0x7febe80a9ac0 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555764945 0 0) 0x7febe81421a0 con 0x7febe813ee90 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.689+0000 7febcffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe8141890 con 0x7febe813ee90 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 10 client.?.objecter ms_handle_connect 0x7febe80af900 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718591029 0 0) 0x7febe8141b70 con 0x7febe80af900 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:18:28.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe81421a0 con 0x7febe80af900 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3957773186 0 0) 0x7febd4004eb0 con 0x7febe80a9ac0 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febe8141b70 con 0x7febe80a9ac0 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.690+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4291267995 0 0) 0x7febe8141890 con 0x7febe813ee90 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febd4004eb0 con 0x7febe813ee90 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7febdc001750 con 0x7febe80a9ac0 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 10 monclient(hunting): dump: 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:28.673 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7febd8001730 con 0x7febe813ee90 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 10 monclient(hunting): dump: 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.674 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2081202159 0 0) 0x7febe81421a0 con 0x7febe80af900 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febe8141890 con 0x7febe80af900 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7febe00018d0 con 0x7febe80af900 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.691+0000 7febcffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 10 monclient(hunting): dump: 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.675 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3879601402 0 0) 0x7febe8141890 con 0x7febe80af900 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 >> v1:172.21.15.157:6790/0 conn(0x7febe813ee90 legacy=0x7febe81de740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 >> v1:172.21.15.157:6789/0 conn(0x7febe80a9ac0 legacy=0x7febe814e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 1 monclient: found mon.b 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe814ea30 con 0x7febe80af900 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 10 monclient: _finish_auth 0 2024-03-29T11:18:28.676 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febed86d640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febed86d640 10 monclient: _renew_subs 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febed86d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febed86d640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe813ff40 con 0x7febe80af900 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febed86d640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febed86d640 10 monclient: _renew_subs 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febed86d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febed86d640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febe81df030 con 0x7febe80af900 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febed86d640 1 librados: init done 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febe0001a30 con 0x7febe80af900 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febcffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.692+0000 7febe57fa640 4 set_mon_vals no callback set 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.693+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7febe0005110 con 0x7febe80af900 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.693+0000 7febcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:28.677 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.693+0000 7febcffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.693+0000 7febcffff640 10 monclient: dump: 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:28.678 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.695+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7febe001b0e0 con 0x7febe80af900 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.696+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7febe0084d30 con 0x7febe80af900 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.696+0000 7febcffff640 10 client.4465.objecter ms_dispatch 0x7febe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.696+0000 7febcffff640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.696+0000 7febcffff640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.696+0000 7febe6ffd640 10 monclient: get_auth_request con 0x7febd406e150 auth_method 0 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.696+0000 7febcffff640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.696+0000 7febcffff640 10 client.4465.objecter ms_handle_connect 0x7febd406e150 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.697+0000 7febed86d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.697+0000 7febed86d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.697+0000 7febed86d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:28.680 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.697+0000 7febed86d640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc8005080 con 0x7febe80af900 2024-03-29T11:18:28.684 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.707+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febe0050610 con 0x7febe80af900 2024-03-29T11:18:28.685 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.707+0000 7febcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:28.685 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:28.707+0000 7febcffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:29.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.023+0000 7febed86d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:29.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.023+0000 7febed86d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:29.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.023+0000 7febed86d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:29.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.023+0000 7febed86d640 1 -- 172.21.15.157:0/1655165562 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febc8005330 con 0x7febe80af900 2024-03-29T11:18:29.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.025+0000 7febcffff640 1 -- 172.21.15.157:0/1655165562 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7febe0052540 con 0x7febe80af900 2024-03-29T11:18:29.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.025+0000 7febcffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:29.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.025+0000 7febcffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:29.003 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:18:29.003 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:29.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.028+0000 7febed86d640 10 librados: watch_flush enter 2024-03-29T11:18:29.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.028+0000 7febed86d640 10 librados: watch_flush exit 2024-03-29T11:18:29.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.028+0000 7febed86d640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-29T11:18:29.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.028+0000 7febed86d640 10 client.4465.objecter successfully canceled tick 2024-03-29T11:18:29.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.029+0000 7febed86d640 1 -- 172.21.15.157:0/1655165562 >> v1:172.21.15.157:6816/1747058247 conn(0x7febd406e150 legacy=0x7febd4070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:29.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.029+0000 7febed86d640 10 monclient: shutdown 2024-03-29T11:18:29.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.029+0000 7febed86d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:29.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.029+0000 7febed86d640 1 -- 172.21.15.157:0/1655165562 >> v1:172.21.15.195:6789/0 conn(0x7febe80af900 legacy=0x7febe813f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:29.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.029+0000 7febed86d640 1 -- 172.21.15.157:0/1655165562 shutdown_connections 2024-03-29T11:18:29.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.029+0000 7febed86d640 1 -- 172.21.15.157:0/1655165562 >> 172.21.15.157:0/1655165562 conn(0x7febe80b35e0 msgr2=0x7febe8132f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:29.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.030+0000 7febed86d640 1 -- 172.21.15.157:0/1655165562 shutdown_connections 2024-03-29T11:18:29.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.031+0000 7febed86d640 1 -- 172.21.15.157:0/1655165562 wait complete. 2024-03-29T11:18:29.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:29.031+0000 7febed86d640 1 librados: shutdown 2024-03-29T11:18:29.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:35.831 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:35.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.006+0000 7f0636455640 10 monclient: get_monmap_and_config 2024-03-29T11:18:35.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.006+0000 7f0636455640 10 monclient: build_initial_monmap 2024-03-29T11:18:35.983 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.006+0000 7f0636455640 10 monclient: monmap: 2024-03-29T11:18:35.984 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:35.984 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:35.984 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:18:36.007512+0000 2024-03-29T11:18:35.984 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:18:36.007512+0000 2024-03-29T11:18:35.984 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:35.984 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:35.984 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:35.984 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:18:35.984 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:18:35.984 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:35.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.007+0000 7f0636455640 1 Processor -- start 2024-03-29T11:18:35.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.007+0000 7f0636455640 1 -- start start 2024-03-29T11:18:35.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.007+0000 7f0636455640 10 monclient: init 2024-03-29T11:18:35.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.008+0000 7f0636455640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:35.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.008+0000 7f0636455640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:35.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.008+0000 7f0636455640 10 monclient(hunting): picked mon.noname-a con 0x7f06301422b0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:35.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.008+0000 7f0636455640 10 monclient(hunting): picked mon.noname-b con 0x7f0630142690 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:35.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.008+0000 7f0636455640 10 monclient(hunting): picked mon.noname-c con 0x7f06300b17d0 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:35.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.008+0000 7f0636455640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0630142c80 con 0x7f06301422b0 2024-03-29T11:18:35.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.008+0000 7f0636455640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06300aae40 con 0x7f06300b17d0 2024-03-29T11:18:35.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.008+0000 7f0636455640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063013fbd0 con 0x7f0630142690 2024-03-29T11:18:35.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.008+0000 7f0636455640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:35.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.008+0000 7f0636455640 10 monclient(hunting): authenticate will time out at 1328.205339s 2024-03-29T11:18:35.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.010+0000 7f062f7fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f0630142690 0x7f06300b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:34408/0 (socket says 172.21.15.157:34408) 2024-03-29T11:18:35.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.010+0000 7f062f7fe640 1 -- 172.21.15.157:0/2566724280 learned_addr learned my addr 172.21.15.157:0/2566724280 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:35.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.010+0000 7f062effd640 1 -- 172.21.15.157:0/2566724280 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454825134 0 0) 0x7f063013fbd0 con 0x7f0630142690 2024-03-29T11:18:35.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.010+0000 7f062effd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:35.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.010+0000 7f062effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:35.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.010+0000 7f062effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:35.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.010+0000 7f062effd640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:18:35.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.010+0000 7f062effd640 1 -- 172.21.15.157:0/2566724280 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f061c003520 con 0x7f0630142690 2024-03-29T11:18:35.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.011+0000 7f062effd640 1 -- 172.21.15.157:0/2566724280 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 53758224 0 0) 0x7f061c003520 con 0x7f0630142690 2024-03-29T11:18:35.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.011+0000 7f062effd640 1 -- 172.21.15.157:0/2566724280 >> v1:172.21.15.195:6789/0 conn(0x7f06300b17d0 legacy=0x7f06300b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:35.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.011+0000 7f062effd640 1 -- 172.21.15.157:0/2566724280 >> v1:172.21.15.157:6789/0 conn(0x7f06301422b0 legacy=0x7f063013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:35.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 1 monclient: found mon.noname-b 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 1 -- 172.21.15.157:0/2566724280 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0630151940 con 0x7f0630142690 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 10 monclient: _finish_auth 0 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 1 -- 172.21.15.157:0/2566724280 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f0618001690 con 0x7f0630142690 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.012+0000 7f062effd640 10 monclient: dump: 2024-03-29T11:18:35.990 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:35.991 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:35.991 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:35.991 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:35.991 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:35.991 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:35.991 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:35.991 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:35.991 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:35.991 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:35.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.013+0000 7f0636455640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:18:35.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.013+0000 7f0636455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:35.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.013+0000 7f062effd640 1 -- 172.21.15.157:0/2566724280 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06180017f0 con 0x7f0630142690 2024-03-29T11:18:35.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.013+0000 7f062effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:35.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.013+0000 7f062effd640 1 -- 172.21.15.157:0/2566724280 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f0618001c40 con 0x7f0630142690 2024-03-29T11:18:35.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.013+0000 7f062effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.013+0000 7f062effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.013+0000 7f062effd640 10 monclient: dump: 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.013+0000 7f0636455640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:35.993 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.013+0000 7f0636455640 4 set_mon_vals no callback set 2024-03-29T11:18:35.994 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.014+0000 7f0636455640 10 monclient: shutdown 2024-03-29T11:18:35.994 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.014+0000 7f0636455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:35.994 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.014+0000 7f0636455640 1 -- 172.21.15.157:0/2566724280 >> v1:172.21.15.157:6790/0 conn(0x7f0630142690 legacy=0x7f06300b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.994 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.014+0000 7f0636455640 1 -- 172.21.15.157:0/2566724280 shutdown_connections 2024-03-29T11:18:35.994 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.014+0000 7f0636455640 1 -- 172.21.15.157:0/2566724280 >> 172.21.15.157:0/2566724280 conn(0x7f0630066a00 msgr2=0x7f0630066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:35.994 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.014+0000 7f0636455640 1 -- 172.21.15.157:0/2566724280 shutdown_connections 2024-03-29T11:18:35.994 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.015+0000 7f0636455640 1 -- 172.21.15.157:0/2566724280 wait complete. 2024-03-29T11:18:35.996 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.015+0000 7f0636455640 10 monclient: build_initial_monmap 2024-03-29T11:18:35.996 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.015+0000 7f0636455640 10 monclient: monmap: 2024-03-29T11:18:35.996 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:35.996 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:35.996 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.016+0000 7f0636455640 1 librados: starting msgr at 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.016+0000 7f0636455640 1 librados: starting objecter 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.016+0000 7f0636455640 1 Processor -- start 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.016+0000 7f0636455640 1 -- start start 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.016+0000 7f0636455640 1 librados: setting wanted keys 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.016+0000 7f0636455640 1 librados: calling monclient init 2024-03-29T11:18:35.997 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.016+0000 7f0636455640 10 monclient: init 2024-03-29T11:18:35.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.017+0000 7f0636455640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:35.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.017+0000 7f0636455640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:35.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.017+0000 7f0636455640 10 monclient(hunting): picked mon.noname-b con 0x7f06300b17d0 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:35.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.017+0000 7f0636455640 10 monclient(hunting): picked mon.noname-a con 0x7f06301422b0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:35.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.017+0000 7f0636455640 10 monclient(hunting): picked mon.noname-c con 0x7f0630142690 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:35.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.017+0000 7f0636455640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063006f040 con 0x7f06301422b0 2024-03-29T11:18:35.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.017+0000 7f0636455640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063006f2b0 con 0x7f06300b17d0 2024-03-29T11:18:35.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.017+0000 7f0636455640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0630131dc0 con 0x7f0630142690 2024-03-29T11:18:35.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.017+0000 7f0636455640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:35.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.017+0000 7f0636455640 10 monclient(hunting): authenticate will time out at 1328.214446s 2024-03-29T11:18:35.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.019+0000 7f06349f2640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f0630142690 0x7f06300711b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:37268/0 (socket says 172.21.15.157:37268) 2024-03-29T11:18:35.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.019+0000 7f06349f2640 1 -- 172.21.15.157:0/2665316803 learned_addr learned my addr 172.21.15.157:0/2665316803 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:35.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.019+0000 7f062cff9640 10 client.?.objecter ms_handle_connect 0x7f0630142690 2024-03-29T11:18:35.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.019+0000 7f062cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:35.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.019+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340563809 0 0) 0x7f0630131dc0 con 0x7f0630142690 2024-03-29T11:18:35.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.019+0000 7f062cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:18:35.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.019+0000 7f062cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:35.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.019+0000 7f062cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:35.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.019+0000 7f062cff9640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:18:35.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.020+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0610004eb0 con 0x7f0630142690 2024-03-29T11:18:36.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.023+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 432698551 0 0) 0x7f0610004eb0 con 0x7f0630142690 2024-03-29T11:18:36.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.023+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0630131dc0 con 0x7f0630142690 2024-03-29T11:18:36.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.023+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f0620001b70 con 0x7f0630142690 2024-03-29T11:18:36.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.023+0000 7f062cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:36.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.023+0000 7f062cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:36.008 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.023+0000 7f062cff9640 10 monclient(hunting): dump: 2024-03-29T11:18:36.008 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:36.008 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:36.008 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:36.008 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:36.008 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:36.008 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:36.009 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:36.009 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:36.009 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:36.009 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:36.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.023+0000 7f062cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:36.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 client.?.objecter ms_handle_connect 0x7f06301422b0 2024-03-29T11:18:36.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:36.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 client.?.objecter ms_handle_connect 0x7f06300b17d0 2024-03-29T11:18:36.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:36.009 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156437329 0 0) 0x7f063006f040 con 0x7f06301422b0 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0610004eb0 con 0x7f06301422b0 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331908850 0 0) 0x7f063006f2b0 con 0x7f06300b17d0 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f063006f040 con 0x7f06300b17d0 2024-03-29T11:18:36.010 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3057545308 0 0) 0x7f0610004eb0 con 0x7f06301422b0 2024-03-29T11:18:36.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f063006f2b0 con 0x7f06301422b0 2024-03-29T11:18:36.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f0618001b00 con 0x7f06301422b0 2024-03-29T11:18:36.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:36.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:36.011 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.024+0000 7f062cff9640 10 monclient(hunting): dump: 2024-03-29T11:18:36.011 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:36.011 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:36.011 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:36.011 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:36.011 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2815821636 0 0) 0x7f063006f040 con 0x7f06300b17d0 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0610004eb0 con 0x7f06300b17d0 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f0628001510 con 0x7f06300b17d0 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 10 monclient(hunting): dump: 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:36.012 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:36.013 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:36.013 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:36.013 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:36.013 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:36.013 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:36.013 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:36.013 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:36.013 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:36.013 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 693066092 0 0) 0x7f0610004eb0 con 0x7f06300b17d0 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 >> v1:172.21.15.157:6790/0 conn(0x7f0630142690 legacy=0x7f06300711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 >> v1:172.21.15.157:6789/0 conn(0x7f06301422b0 legacy=0x7f0630070aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 1 monclient: found mon.b 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0630131a90 con 0x7f06300b17d0 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 10 monclient: _finish_auth 0 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f062cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f0636455640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:18:36.014 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f0636455640 10 monclient: _renew_subs 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f0636455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.025+0000 7f0636455640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0630132a10 con 0x7f06300b17d0 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.026+0000 7f0636455640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.026+0000 7f0636455640 10 monclient: _renew_subs 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.026+0000 7f0636455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.026+0000 7f0636455640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0630071aa0 con 0x7f06300b17d0 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.026+0000 7f0636455640 1 librados: init done 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f0636455640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f0636455640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f0636455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f0636455640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0630075f10 con 0x7f06300b17d0 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0628001670 con 0x7f06300b17d0 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f062cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:36.015 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f0628001ce0 con 0x7f06300b17d0 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f062cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f062cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f062cff9640 10 monclient: dump: 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:36.016 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.027+0000 7f062e7fc640 4 set_mon_vals no callback set 2024-03-29T11:18:36.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.029+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f0628019700 con 0x7f06300b17d0 2024-03-29T11:18:36.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.030+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f0628083720 con 0x7f06300b17d0 2024-03-29T11:18:36.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.030+0000 7f062cff9640 10 client.4468.objecter ms_dispatch 0x7f0630000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:36.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.030+0000 7f062cff9640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:36.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.030+0000 7f062cff9640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:36.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.030+0000 7f062cff9640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-29T11:18:36.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.030+0000 7f062f7fe640 10 monclient: get_auth_request con 0x7f061006e160 auth_method 0 2024-03-29T11:18:36.017 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.030+0000 7f062cff9640 10 client.4468.objecter ms_handle_connect 0x7f061006e160 2024-03-29T11:18:36.018 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.038+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f062804ee30 con 0x7f06300b17d0 2024-03-29T11:18:36.018 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.038+0000 7f062cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:36.018 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.038+0000 7f062cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:36.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.358+0000 7f0636455640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:36.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.358+0000 7f0636455640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:36.335 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.358+0000 7f0636455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:36.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.358+0000 7f0636455640 1 -- 172.21.15.157:0/2665316803 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0630131c20 con 0x7f06300b17d0 2024-03-29T11:18:36.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.361+0000 7f062cff9640 1 -- 172.21.15.157:0/2665316803 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f0630131c20 con 0x7f06300b17d0 2024-03-29T11:18:36.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.361+0000 7f062cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:36.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.361+0000 7f062cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:36.339 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:18:36.339 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:36.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.365+0000 7f0636455640 10 librados: watch_flush enter 2024-03-29T11:18:36.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.365+0000 7f0636455640 10 librados: watch_flush exit 2024-03-29T11:18:36.342 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.365+0000 7f0636455640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-29T11:18:36.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.365+0000 7f0636455640 10 client.4468.objecter successfully canceled tick 2024-03-29T11:18:36.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.365+0000 7f0636455640 1 -- 172.21.15.157:0/2665316803 >> v1:172.21.15.157:6816/1747058247 conn(0x7f061006e160 legacy=0x7f0610070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:36.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.365+0000 7f0636455640 10 monclient: shutdown 2024-03-29T11:18:36.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.365+0000 7f0636455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:36.343 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.365+0000 7f0636455640 1 -- 172.21.15.157:0/2665316803 >> v1:172.21.15.195:6789/0 conn(0x7f06300b17d0 legacy=0x7f0630132300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:36.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.367+0000 7f0636455640 1 -- 172.21.15.157:0/2665316803 shutdown_connections 2024-03-29T11:18:36.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.367+0000 7f0636455640 1 -- 172.21.15.157:0/2665316803 >> 172.21.15.157:0/2665316803 conn(0x7f0630066a00 msgr2=0x7f06300b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:36.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.367+0000 7f0636455640 1 -- 172.21.15.157:0/2665316803 shutdown_connections 2024-03-29T11:18:36.344 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.367+0000 7f0636455640 1 -- 172.21.15.157:0/2665316803 wait complete. 2024-03-29T11:18:36.345 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:36.368+0000 7f0636455640 1 librados: shutdown 2024-03-29T11:18:36.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:43.170 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:43.327 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.350+0000 7f7d68ff4640 10 monclient: get_monmap_and_config 2024-03-29T11:18:43.327 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.350+0000 7f7d68ff4640 10 monclient: build_initial_monmap 2024-03-29T11:18:43.327 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.350+0000 7f7d68ff4640 10 monclient: monmap: 2024-03-29T11:18:43.327 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:43.327 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:43.328 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:18:43.351146+0000 2024-03-29T11:18:43.328 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:18:43.351146+0000 2024-03-29T11:18:43.328 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:43.328 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:43.328 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:43.328 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:18:43.328 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:18:43.328 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:43.329 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.351+0000 7f7d68ff4640 1 Processor -- start 2024-03-29T11:18:43.329 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.351+0000 7f7d68ff4640 1 -- start start 2024-03-29T11:18:43.329 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.351+0000 7f7d68ff4640 10 monclient: init 2024-03-29T11:18:43.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.352+0000 7f7d68ff4640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:43.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.352+0000 7f7d68ff4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:43.330 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.352+0000 7f7d68ff4640 10 monclient(hunting): picked mon.noname-a con 0x7f7d641422b0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:43.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.352+0000 7f7d68ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f7d64142690 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:43.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.352+0000 7f7d68ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f7d640b17d0 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:43.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.352+0000 7f7d68ff4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d64142c80 con 0x7f7d641422b0 2024-03-29T11:18:43.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.352+0000 7f7d68ff4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d640aae40 con 0x7f7d64142690 2024-03-29T11:18:43.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.352+0000 7f7d68ff4640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6413fbd0 con 0x7f7d640b17d0 2024-03-29T11:18:43.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.352+0000 7f7d68ff4640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:43.331 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.352+0000 7f7d68ff4640 10 monclient(hunting): authenticate will time out at 1335.549519s 2024-03-29T11:18:43.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.357+0000 7f7d62d9d640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f7d640b17d0 0x7f7d640b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:41009/0 (socket says 172.21.15.157:41009) 2024-03-29T11:18:43.334 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.357+0000 7f7d62d9d640 1 -- 172.21.15.157:0/704301059 learned_addr learned my addr 172.21.15.157:0/704301059 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.357+0000 7f7d6159a640 1 -- 172.21.15.157:0/704301059 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637142913 0 0) 0x7f7d640aae40 con 0x7f7d64142690 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.357+0000 7f7d6159a640 10 monclient(hunting): _negotiate 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.357+0000 7f7d6159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.357+0000 7f7d6159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.357+0000 7f7d6159a640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.357+0000 7f7d6159a640 1 -- 172.21.15.157:0/704301059 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c003520 con 0x7f7d64142690 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.358+0000 7f7d6159a640 1 -- 172.21.15.157:0/704301059 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202311359 0 0) 0x7f7d6413fbd0 con 0x7f7d640b17d0 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.358+0000 7f7d6159a640 10 monclient(hunting): _negotiate 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.358+0000 7f7d6159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.358+0000 7f7d6159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.358+0000 7f7d6159a640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:18:43.336 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.358+0000 7f7d6159a640 1 -- 172.21.15.157:0/704301059 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d640aae40 con 0x7f7d640b17d0 2024-03-29T11:18:43.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.358+0000 7f7d6159a640 1 -- 172.21.15.157:0/704301059 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1675491956 0 0) 0x7f7d4c003520 con 0x7f7d64142690 2024-03-29T11:18:43.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.358+0000 7f7d6159a640 1 -- 172.21.15.157:0/704301059 >> v1:172.21.15.157:6790/0 conn(0x7f7d640b17d0 legacy=0x7f7d640b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:43.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.358+0000 7f7d6159a640 1 -- 172.21.15.157:0/704301059 >> v1:172.21.15.157:6789/0 conn(0x7f7d641422b0 legacy=0x7f7d6413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:43.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 10 monclient: _finish_hunting 0 2024-03-29T11:18:43.337 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 1 monclient: found mon.noname-c 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 1 -- 172.21.15.157:0/704301059 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d64151940 con 0x7f7d64142690 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 10 monclient: _finish_auth 0 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 10 monclient: _check_auth_tickets 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 1 -- 172.21.15.157:0/704301059 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7d54001690 con 0x7f7d64142690 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 10 monclient: dump: 2024-03-29T11:18:43.338 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d68ff4640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d68ff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 1 -- 172.21.15.157:0/704301059 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d540017f0 con 0x7f7d64142690 2024-03-29T11:18:43.339 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d6159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:43.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d68ff4640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:43.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.359+0000 7f7d68ff4640 4 set_mon_vals no callback set 2024-03-29T11:18:43.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.360+0000 7f7d68ff4640 10 monclient: shutdown 2024-03-29T11:18:43.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.360+0000 7f7d68ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:43.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.360+0000 7f7d68ff4640 1 -- 172.21.15.157:0/704301059 >> v1:172.21.15.195:6789/0 conn(0x7f7d64142690 legacy=0x7f7d640b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:43.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.361+0000 7f7d68ff4640 1 -- 172.21.15.157:0/704301059 shutdown_connections 2024-03-29T11:18:43.340 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.361+0000 7f7d68ff4640 1 -- 172.21.15.157:0/704301059 >> 172.21.15.157:0/704301059 conn(0x7f7d64066a00 msgr2=0x7f7d64066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:43.341 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.361+0000 7f7d68ff4640 1 -- 172.21.15.157:0/704301059 shutdown_connections 2024-03-29T11:18:43.341 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.362+0000 7f7d68ff4640 1 -- 172.21.15.157:0/704301059 wait complete. 2024-03-29T11:18:43.346 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.363+0000 7f7d68ff4640 10 monclient: build_initial_monmap 2024-03-29T11:18:43.346 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.363+0000 7f7d68ff4640 10 monclient: monmap: 2024-03-29T11:18:43.346 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:43.346 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:43.346 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:18:43.346 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:18:43.346 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:43.346 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.363+0000 7f7d68ff4640 1 librados: starting msgr at 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.363+0000 7f7d68ff4640 1 librados: starting objecter 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.363+0000 7f7d68ff4640 1 Processor -- start 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.363+0000 7f7d68ff4640 1 -- start start 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.363+0000 7f7d68ff4640 1 librados: setting wanted keys 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.363+0000 7f7d68ff4640 1 librados: calling monclient init 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.363+0000 7f7d68ff4640 10 monclient: init 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.364+0000 7f7d68ff4640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:43.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.364+0000 7f7d68ff4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.364+0000 7f7d68ff4640 10 monclient(hunting): picked mon.noname-a con 0x7f7d640b17d0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.364+0000 7f7d68ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f7d641422b0 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.364+0000 7f7d68ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f7d64142690 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.364+0000 7f7d68ff4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d640ab9b0 con 0x7f7d640b17d0 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.364+0000 7f7d68ff4640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d640ac760 con 0x7f7d641422b0 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.364+0000 7f7d68ff4640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d641db720 con 0x7f7d64142690 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.364+0000 7f7d68ff4640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.364+0000 7f7d68ff4640 10 monclient(hunting): authenticate will time out at 1335.561550s 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d61d9b640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7d641422b0 0x7f7d641da900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:40981/0 (socket says 172.21.15.157:40981) 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d61d9b640 1 -- 172.21.15.157:0/1686051518 learned_addr learned my addr 172.21.15.157:0/1686051518 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d52ffd640 10 client.?.objecter ms_handle_connect 0x7f7d641422b0 2024-03-29T11:18:43.348 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:43.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495310342 0 0) 0x7f7d640ac760 con 0x7f7d641422b0 2024-03-29T11:18:43.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:43.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d52ffd640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d44004eb0 con 0x7f7d641422b0 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.369+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1240893304 0 0) 0x7f7d44004eb0 con 0x7f7d641422b0 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.370+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d44005300 con 0x7f7d641422b0 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.370+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7d54003c30 con 0x7f7d641422b0 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.370+0000 7f7d52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.370+0000 7f7d52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.370+0000 7f7d52ffd640 10 monclient(hunting): dump: 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:43.350 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:43.351 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:43.351 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:43.351 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:43.351 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:43.351 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:43.351 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:43.351 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:43.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.370+0000 7f7d52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:43.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.370+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2841563959 0 0) 0x7f7d44005300 con 0x7f7d641422b0 2024-03-29T11:18:43.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.370+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 >> v1:172.21.15.157:6790/0 conn(0x7f7d64142690 legacy=0x7f7d641db010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:43.352 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.370+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 >> v1:172.21.15.157:6789/0 conn(0x7f7d640b17d0 legacy=0x7f7d640b5780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:43.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d52ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:43.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d52ffd640 1 monclient: found mon.b 2024-03-29T11:18:43.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:43.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d52ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:43.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d640ac9d0 con 0x7f7d641422b0 2024-03-29T11:18:43.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d52ffd640 10 monclient: _finish_auth 0 2024-03-29T11:18:43.353 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d52ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d68ff4640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d68ff4640 10 monclient: _renew_subs 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d68ff4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d68ff4640 1 -- 172.21.15.157:0/1686051518 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d640b6a40 con 0x7f7d641422b0 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d68ff4640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d68ff4640 10 monclient: _renew_subs 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d68ff4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d68ff4640 1 -- 172.21.15.157:0/1686051518 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d640b78d0 con 0x7f7d641422b0 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.374+0000 7f7d68ff4640 1 librados: init done 2024-03-29T11:18:43.354 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.375+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d54001a20 con 0x7f7d641422b0 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.375+0000 7f7d52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.375+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7d54003c30 con 0x7f7d641422b0 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.375+0000 7f7d52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.375+0000 7f7d52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.375+0000 7f7d52ffd640 10 monclient: dump: 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:43.355 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:43.356 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:43.356 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:43.356 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:43.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.375+0000 7f7d60d99640 4 set_mon_vals no callback set 2024-03-29T11:18:43.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.377+0000 7f7d68ff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:43.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.377+0000 7f7d68ff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:43.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.377+0000 7f7d68ff4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:43.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.377+0000 7f7d68ff4640 1 -- 172.21.15.157:0/1686051518 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d48005080 con 0x7f7d641422b0 2024-03-29T11:18:43.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.377+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f7d54003e90 con 0x7f7d641422b0 2024-03-29T11:18:43.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.378+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f7d54084a30 con 0x7f7d641422b0 2024-03-29T11:18:43.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.378+0000 7f7d52ffd640 10 client.4474.objecter ms_dispatch 0x7f7d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:43.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.378+0000 7f7d52ffd640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:43.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.378+0000 7f7d52ffd640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:43.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.378+0000 7f7d52ffd640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T11:18:43.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.378+0000 7f7d6259c640 10 monclient: get_auth_request con 0x7f7d4406dfc0 auth_method 0 2024-03-29T11:18:43.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.378+0000 7f7d52ffd640 10 client.4474.objecter ms_handle_connect 0x7f7d4406dfc0 2024-03-29T11:18:43.364 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.387+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d54050310 con 0x7f7d641422b0 2024-03-29T11:18:43.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.387+0000 7f7d52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:43.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.387+0000 7f7d52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:43.693 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.715+0000 7f7d68ff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:43.693 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.715+0000 7f7d68ff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:43.693 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.715+0000 7f7d68ff4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:43.693 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.715+0000 7f7d68ff4640 1 -- 172.21.15.157:0/1686051518 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d48005330 con 0x7f7d641422b0 2024-03-29T11:18:43.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.720+0000 7f7d52ffd640 1 -- 172.21.15.157:0/1686051518 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f7d54052240 con 0x7f7d641422b0 2024-03-29T11:18:43.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.720+0000 7f7d52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:43.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.720+0000 7f7d52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:43.698 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:18:43.698 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:43.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.727+0000 7f7d68ff4640 10 librados: watch_flush enter 2024-03-29T11:18:43.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.727+0000 7f7d68ff4640 10 librados: watch_flush exit 2024-03-29T11:18:43.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.727+0000 7f7d68ff4640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T11:18:43.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.727+0000 7f7d68ff4640 10 client.4474.objecter successfully canceled tick 2024-03-29T11:18:43.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.728+0000 7f7d68ff4640 1 -- 172.21.15.157:0/1686051518 >> v1:172.21.15.157:6816/1747058247 conn(0x7f7d4406dfc0 legacy=0x7f7d44070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:43.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.728+0000 7f7d68ff4640 10 monclient: shutdown 2024-03-29T11:18:43.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.728+0000 7f7d68ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:43.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.728+0000 7f7d68ff4640 1 -- 172.21.15.157:0/1686051518 >> v1:172.21.15.195:6789/0 conn(0x7f7d641422b0 legacy=0x7f7d641da900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:43.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.729+0000 7f7d68ff4640 1 -- 172.21.15.157:0/1686051518 shutdown_connections 2024-03-29T11:18:43.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.729+0000 7f7d68ff4640 1 -- 172.21.15.157:0/1686051518 >> 172.21.15.157:0/1686051518 conn(0x7f7d64066a00 msgr2=0x7f7d6412fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:43.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.729+0000 7f7d68ff4640 1 -- 172.21.15.157:0/1686051518 shutdown_connections 2024-03-29T11:18:43.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.729+0000 7f7d68ff4640 1 -- 172.21.15.157:0/1686051518 wait complete. 2024-03-29T11:18:43.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:43.729+0000 7f7d68ff4640 1 librados: shutdown 2024-03-29T11:18:43.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:50.535 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:50.695 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.717+0000 7f76d1e55640 10 monclient: get_monmap_and_config 2024-03-29T11:18:50.695 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.717+0000 7f76d1e55640 10 monclient: build_initial_monmap 2024-03-29T11:18:50.695 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.717+0000 7f76d1e55640 10 monclient: monmap: 2024-03-29T11:18:50.695 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:50.695 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:50.695 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:18:50.718673+0000 2024-03-29T11:18:50.695 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:18:50.718673+0000 2024-03-29T11:18:50.695 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:50.696 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:50.696 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:50.696 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:18:50.696 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:18:50.696 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:50.697 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.720+0000 7f76d1e55640 1 Processor -- start 2024-03-29T11:18:50.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.720+0000 7f76d1e55640 1 -- start start 2024-03-29T11:18:50.698 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.720+0000 7f76d1e55640 10 monclient: init 2024-03-29T11:18:50.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.721+0000 7f76d1e55640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:50.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.721+0000 7f76d1e55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:50.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.721+0000 7f76d1e55640 10 monclient(hunting): picked mon.noname-b con 0x7f76cc0b0770 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:50.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.721+0000 7f76d1e55640 10 monclient(hunting): picked mon.noname-c con 0x7f76cc14a700 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:50.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.721+0000 7f76d1e55640 10 monclient(hunting): picked mon.noname-a con 0x7f76cc0ac5a0 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:50.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.721+0000 7f76d1e55640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc0aba70 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.721+0000 7f76d1e55640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc0aecb0 con 0x7f76cc14a700 2024-03-29T11:18:50.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.721+0000 7f76d1e55640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc0af150 con 0x7f76cc0b0770 2024-03-29T11:18:50.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.721+0000 7f76d1e55640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:50.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.721+0000 7f76d1e55640 10 monclient(hunting): authenticate will time out at 1342.918556s 2024-03-29T11:18:50.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.725+0000 7f76cbfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f76cc0ac5a0 0x7f76cc151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:56751/0 (socket says 172.21.15.157:56751) 2024-03-29T11:18:50.703 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.725+0000 7f76cbfff640 1 -- 172.21.15.157:0/4139050720 learned_addr learned my addr 172.21.15.157:0/4139050720 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:50.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 1 -- 172.21.15.157:0/4139050720 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170442712 0 0) 0x7f76cc0aecb0 con 0x7f76cc14a700 2024-03-29T11:18:50.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:50.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:50.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:50.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:18:50.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 1 -- 172.21.15.157:0/4139050720 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b4003520 con 0x7f76cc14a700 2024-03-29T11:18:50.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 1 -- 172.21.15.157:0/4139050720 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3652809120 0 0) 0x7f76b4003520 con 0x7f76cc14a700 2024-03-29T11:18:50.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 1 -- 172.21.15.157:0/4139050720 >> v1:172.21.15.157:6790/0 conn(0x7f76cc0b0770 legacy=0x7f76cc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 1 -- 172.21.15.157:0/4139050720 >> v1:172.21.15.157:6789/0 conn(0x7f76cc0ac5a0 legacy=0x7f76cc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 1 monclient: found mon.noname-c 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 1 -- 172.21.15.157:0/4139050720 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc0ac9d0 con 0x7f76cc14a700 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 1 -- 172.21.15.157:0/4139050720 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f76c00016b0 con 0x7f76cc14a700 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:50.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.726+0000 7f76ca7fc640 10 monclient: dump: 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.727+0000 7f76ca7fc640 1 -- 172.21.15.157:0/4139050720 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76c0001880 con 0x7f76cc14a700 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.727+0000 7f76ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:50.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.727+0000 7f76ca7fc640 1 -- 172.21.15.157:0/4139050720 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f76c0001cd0 con 0x7f76cc14a700 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.727+0000 7f76ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.727+0000 7f76ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.727+0000 7f76ca7fc640 10 monclient: dump: 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:50.710 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:50.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.729+0000 7f76d1e55640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:18:50.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.729+0000 7f76d1e55640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:50.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.729+0000 7f76d1e55640 4 set_mon_vals no callback set 2024-03-29T11:18:50.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.730+0000 7f76d1e55640 10 monclient: shutdown 2024-03-29T11:18:50.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.730+0000 7f76d1e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:50.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.730+0000 7f76d1e55640 1 -- 172.21.15.157:0/4139050720 >> v1:172.21.15.195:6789/0 conn(0x7f76cc14a700 legacy=0x7f76cc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:50.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.731+0000 7f76d1e55640 1 -- 172.21.15.157:0/4139050720 shutdown_connections 2024-03-29T11:18:50.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.731+0000 7f76d1e55640 1 -- 172.21.15.157:0/4139050720 >> 172.21.15.157:0/4139050720 conn(0x7f76cc06b660 msgr2=0x7f76cc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:50.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.731+0000 7f76d1e55640 1 -- 172.21.15.157:0/4139050720 shutdown_connections 2024-03-29T11:18:50.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.734+0000 7f76d1e55640 1 -- 172.21.15.157:0/4139050720 wait complete. 2024-03-29T11:18:50.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.735+0000 7f76d1e55640 10 monclient: build_initial_monmap 2024-03-29T11:18:50.712 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.735+0000 7f76d1e55640 10 monclient: monmap: 2024-03-29T11:18:50.712 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.735+0000 7f76d1e55640 1 librados: starting msgr at 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.735+0000 7f76d1e55640 1 librados: starting objecter 2024-03-29T11:18:50.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.736+0000 7f76d1e55640 1 Processor -- start 2024-03-29T11:18:50.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.736+0000 7f76d1e55640 1 -- start start 2024-03-29T11:18:50.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.736+0000 7f76d1e55640 1 librados: setting wanted keys 2024-03-29T11:18:50.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.736+0000 7f76d1e55640 1 librados: calling monclient init 2024-03-29T11:18:50.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.736+0000 7f76d1e55640 10 monclient: init 2024-03-29T11:18:50.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.737+0000 7f76d1e55640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:50.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.737+0000 7f76d1e55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:50.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.737+0000 7f76d1e55640 10 monclient(hunting): picked mon.noname-b con 0x7f76cc0ac5a0 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:50.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.737+0000 7f76d1e55640 10 monclient(hunting): picked mon.noname-c con 0x7f76cc0b0770 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:50.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.737+0000 7f76d1e55640 10 monclient(hunting): picked mon.noname-a con 0x7f76cc14a700 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:50.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.737+0000 7f76d1e55640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc1239d0 con 0x7f76cc14a700 2024-03-29T11:18:50.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.737+0000 7f76d1e55640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc127f30 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.737+0000 7f76d1e55640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc125b40 con 0x7f76cc0b0770 2024-03-29T11:18:50.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.737+0000 7f76d1e55640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:50.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.737+0000 7f76d1e55640 10 monclient(hunting): authenticate will time out at 1342.934154s 2024-03-29T11:18:50.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.740+0000 7f76cb7fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f76cc0ac5a0 0x7f76cc1265b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:53435/0 (socket says 172.21.15.157:53435) 2024-03-29T11:18:50.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.740+0000 7f76cb7fe640 1 -- 172.21.15.157:0/1289281101 learned_addr learned my addr 172.21.15.157:0/1289281101 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:50.717 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.740+0000 7f76b3fff640 10 client.?.objecter ms_handle_connect 0x7f76cc0ac5a0 2024-03-29T11:18:50.718 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.740+0000 7f76b3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:50.719 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.741+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704904072 0 0) 0x7f76cc127f30 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.719 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.741+0000 7f76b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:50.719 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.741+0000 7f76b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.741+0000 7f76b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.741+0000 7f76b3fff640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.741+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b8004eb0 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.742+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1960992034 0 0) 0x7f76b8004eb0 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.742+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76cc127f30 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.742+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f76bc001960 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.742+0000 7f76b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.742+0000 7f76b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.742+0000 7f76b3fff640 10 monclient(hunting): dump: 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:50.720 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:50.721 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:50.721 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:50.721 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:50.721 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:50.721 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:50.721 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:50.721 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.742+0000 7f76b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:50.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.743+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2042734676 0 0) 0x7f76cc127f30 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.743+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 >> v1:172.21.15.157:6790/0 conn(0x7f76cc0b0770 legacy=0x7f76cc144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 >> v1:172.21.15.157:6789/0 conn(0x7f76cc14a700 legacy=0x7f76cc1db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 1 monclient: found mon.b 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc145950 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 10 client.?.objecter ms_handle_connect 0x7f76cc14a700 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.744+0000 7f76b3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76d1e55640 5 monclient: authenticate success, global_id 4480 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76d1e55640 10 monclient: _renew_subs 2024-03-29T11:18:50.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76d1e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76d1e55640 1 -- 172.21.15.157:0/1289281101 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc126cc0 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76bc001c80 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f76bc001960 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76b3fff640 10 monclient: dump: 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:50.726 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76d1e55640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76d1e55640 10 monclient: _renew_subs 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76d1e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.745+0000 7f76d1e55640 1 -- 172.21.15.157:0/1289281101 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc145630 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.746+0000 7f76d1e55640 1 librados: init done 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.746+0000 7f76c9ffb640 4 set_mon_vals no callback set 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.747+0000 7f76d1e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.747+0000 7f76d1e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.747+0000 7f76d1e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:50.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.747+0000 7f76d1e55640 1 -- 172.21.15.157:0/1289281101 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc127070 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.748+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f76bc003920 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.749+0000 7f76caffd640 10 monclient: get_auth_request con 0x7f76b806dec0 auth_method 0 2024-03-29T11:18:50.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.749+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f76bc0544d0 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.749+0000 7f76b3fff640 10 client.4480.objecter ms_dispatch 0x7f76cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:50.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.749+0000 7f76b3fff640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:50.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.749+0000 7f76b3fff640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:50.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.750+0000 7f76b3fff640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-29T11:18:50.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.750+0000 7f76b3fff640 10 client.4480.objecter ms_handle_connect 0x7f76b806dec0 2024-03-29T11:18:50.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.761+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76bc050770 con 0x7f76cc0ac5a0 2024-03-29T11:18:50.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.761+0000 7f76b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:50.739 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:50.761+0000 7f76b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:51.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.086+0000 7f76d1e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:51.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.086+0000 7f76d1e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:51.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.086+0000 7f76d1e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:18:51.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.086+0000 7f76d1e55640 1 -- 172.21.15.157:0/1289281101 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76cc075f10 con 0x7f76cc0ac5a0 2024-03-29T11:18:51.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.089+0000 7f76b3fff640 1 -- 172.21.15.157:0/1289281101 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f76bc050910 con 0x7f76cc0ac5a0 2024-03-29T11:18:51.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.089+0000 7f76b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:51.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.089+0000 7f76b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:51.066 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:18:51.067 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:51.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.092+0000 7f76d1e55640 10 librados: watch_flush enter 2024-03-29T11:18:51.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.095+0000 7f76d1e55640 10 librados: watch_flush exit 2024-03-29T11:18:51.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.095+0000 7f76d1e55640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-29T11:18:51.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.095+0000 7f76d1e55640 10 client.4480.objecter successfully canceled tick 2024-03-29T11:18:51.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.099+0000 7f76d1e55640 1 -- 172.21.15.157:0/1289281101 >> v1:172.21.15.157:6816/1747058247 conn(0x7f76b806dec0 legacy=0x7f76b8070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:51.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.099+0000 7f76d1e55640 10 monclient: shutdown 2024-03-29T11:18:51.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.099+0000 7f76d1e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:51.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.099+0000 7f76d1e55640 1 -- 172.21.15.157:0/1289281101 >> v1:172.21.15.195:6789/0 conn(0x7f76cc0ac5a0 legacy=0x7f76cc1265b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:51.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.099+0000 7f76d1e55640 1 -- 172.21.15.157:0/1289281101 shutdown_connections 2024-03-29T11:18:51.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.099+0000 7f76d1e55640 1 -- 172.21.15.157:0/1289281101 >> 172.21.15.157:0/1289281101 conn(0x7f76cc06b660 msgr2=0x7f76cc0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:51.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.102+0000 7f76d1e55640 1 -- 172.21.15.157:0/1289281101 shutdown_connections 2024-03-29T11:18:51.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.102+0000 7f76d1e55640 1 -- 172.21.15.157:0/1289281101 wait complete. 2024-03-29T11:18:51.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:51.102+0000 7f76d1e55640 1 librados: shutdown 2024-03-29T11:18:51.101 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:57.908 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:58.054 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.077+0000 7fc3b9838640 10 monclient: get_monmap_and_config 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.077+0000 7fc3b9838640 10 monclient: build_initial_monmap 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.077+0000 7fc3b9838640 10 monclient: monmap: 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:18:58.078874+0000 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:18:58.078874+0000 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:18:58.055 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:18:58.056 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:58.056 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.078+0000 7fc3b9838640 1 Processor -- start 2024-03-29T11:18:58.056 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.079+0000 7fc3b9838640 1 -- start start 2024-03-29T11:18:58.056 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.079+0000 7fc3b9838640 10 monclient: init 2024-03-29T11:18:58.058 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.080+0000 7fc3b9838640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:58.058 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.080+0000 7fc3b9838640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:58.058 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.080+0000 7fc3b9838640 10 monclient(hunting): picked mon.noname-a con 0x7fc3ac038a10 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:58.058 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.080+0000 7fc3b9838640 10 monclient(hunting): picked mon.noname-b con 0x7fc3ac038df0 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:58.058 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.080+0000 7fc3b9838640 10 monclient(hunting): picked mon.noname-c con 0x7fc3ac030690 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:58.058 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b9838640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac0c54f0 con 0x7fc3ac038a10 2024-03-29T11:18:58.058 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b9838640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac036460 con 0x7fc3ac030690 2024-03-29T11:18:58.059 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b9838640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac039870 con 0x7fc3ac038df0 2024-03-29T11:18:58.059 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b9838640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:58.059 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b9838640 10 monclient(hunting): authenticate will time out at 1350.278032s 2024-03-29T11:18:58.059 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b37fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc3ac030690 0x7fc3ac0d31a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:51942/0 (socket says 172.21.15.157:51942) 2024-03-29T11:18:58.059 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b37fe640 1 -- 172.21.15.157:0/358417453 learned_addr learned my addr 172.21.15.157:0/358417453 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:58.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924273615 0 0) 0x7fc3ac0c54f0 con 0x7fc3ac038a10 2024-03-29T11:18:58.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b1ffb640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:18:58.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.081+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39c003520 con 0x7fc3ac038a10 2024-03-29T11:18:58.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552201473 0 0) 0x7fc3ac036460 con 0x7fc3ac030690 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac0c54f0 con 0x7fc3ac030690 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1350035342 0 0) 0x7fc39c003520 con 0x7fc3ac038a10 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 >> v1:172.21.15.157:6790/0 conn(0x7fc3ac038df0 legacy=0x7fc3ac0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 >> v1:172.21.15.195:6789/0 conn(0x7fc3ac030690 legacy=0x7fc3ac0d31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:18:58.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac0356c0 con 0x7fc3ac038a10 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc3a4001690 con 0x7fc3ac038a10 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: dump: 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:58.063 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b9838640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b9838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3a4001860 con 0x7fc3ac038a10 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 1 -- 172.21.15.157:0/358417453 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc3a4001c70 con 0x7fc3ac038a10 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.082+0000 7fc3b1ffb640 10 monclient: dump: 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:58.064 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.083+0000 7fc3b9838640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.083+0000 7fc3b9838640 4 set_mon_vals no callback set 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.084+0000 7fc3b9838640 10 monclient: shutdown 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.084+0000 7fc3b9838640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.084+0000 7fc3b9838640 1 -- 172.21.15.157:0/358417453 >> v1:172.21.15.157:6789/0 conn(0x7fc3ac038a10 legacy=0x7fc3ac033550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.084+0000 7fc3b9838640 1 -- 172.21.15.157:0/358417453 shutdown_connections 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.084+0000 7fc3b9838640 1 -- 172.21.15.157:0/358417453 >> 172.21.15.157:0/358417453 conn(0x7fc3ac005c30 msgr2=0x7fc3ac006040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:58.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.084+0000 7fc3b9838640 1 -- 172.21.15.157:0/358417453 shutdown_connections 2024-03-29T11:18:58.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.084+0000 7fc3b9838640 1 -- 172.21.15.157:0/358417453 wait complete. 2024-03-29T11:18:58.072 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.085+0000 7fc3b9838640 10 monclient: build_initial_monmap 2024-03-29T11:18:58.072 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.085+0000 7fc3b9838640 10 monclient: monmap: 2024-03-29T11:18:58.072 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.086+0000 7fc3b9838640 1 librados: starting msgr at 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.086+0000 7fc3b9838640 1 librados: starting objecter 2024-03-29T11:18:58.073 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.086+0000 7fc3b9838640 1 Processor -- start 2024-03-29T11:18:58.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.087+0000 7fc3b9838640 1 -- start start 2024-03-29T11:18:58.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.087+0000 7fc3b9838640 1 librados: setting wanted keys 2024-03-29T11:18:58.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.087+0000 7fc3b9838640 1 librados: calling monclient init 2024-03-29T11:18:58.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.087+0000 7fc3b9838640 10 monclient: init 2024-03-29T11:18:58.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.089+0000 7fc3b9838640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:58.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.089+0000 7fc3b9838640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:58.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.089+0000 7fc3b9838640 10 monclient(hunting): picked mon.noname-a con 0x7fc3ac030690 addr v1:172.21.15.157:6789/0 2024-03-29T11:18:58.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.089+0000 7fc3b9838640 10 monclient(hunting): picked mon.noname-b con 0x7fc3ac038a10 addr v1:172.21.15.195:6789/0 2024-03-29T11:18:58.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.089+0000 7fc3b9838640 10 monclient(hunting): picked mon.noname-c con 0x7fc3ac0c98a0 addr v1:172.21.15.157:6790/0 2024-03-29T11:18:58.074 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.089+0000 7fc3b9838640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac0c6a70 con 0x7fc3ac030690 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.089+0000 7fc3b37fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fc3ac0c98a0 0x7fc3ac15cc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:60226/0 (socket says 172.21.15.157:60226) 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.089+0000 7fc3b37fe640 1 -- 172.21.15.157:0/1166464885 learned_addr learned my addr 172.21.15.157:0/1166464885 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac0a5720 con 0x7fc3ac038a10 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac0ca950 con 0x7fc3ac0c98a0 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc3b9838640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc3b9838640 10 monclient(hunting): authenticate will time out at 1350.286909s 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 10 client.?.objecter ms_handle_connect 0x7fc3ac038a10 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 10 client.?.objecter ms_handle_connect 0x7fc3ac0c98a0 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 10 client.?.objecter ms_handle_connect 0x7fc3ac030690 2024-03-29T11:18:58.075 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372573996 0 0) 0x7fc3ac0a5720 con 0x7fc3ac038a10 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.090+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a0004eb0 con 0x7fc3ac038a10 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554963420 0 0) 0x7fc3ac0c6a70 con 0x7fc3ac030690 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:18:58.076 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac0a5720 con 0x7fc3ac030690 2024-03-29T11:18:58.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894097249 0 0) 0x7fc3ac0ca950 con 0x7fc3ac0c98a0 2024-03-29T11:18:58.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:58.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:58.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:58.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:18:58.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac0c6a70 con 0x7fc3ac0c98a0 2024-03-29T11:18:58.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1949487357 0 0) 0x7fc3a0004eb0 con 0x7fc3ac038a10 2024-03-29T11:18:58.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3ac0ca950 con 0x7fc3ac038a10 2024-03-29T11:18:58.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc3a8001820 con 0x7fc3ac038a10 2024-03-29T11:18:58.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.078 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:58.078 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): dump: 2024-03-29T11:18:58.078 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:58.078 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:58.078 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:58.078 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:58.078 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.078 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:58.078 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:58.078 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2393268083 0 0) 0x7fc3ac0a5720 con 0x7fc3ac030690 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3a0004eb0 con 0x7fc3ac030690 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc3a4001b70 con 0x7fc3ac030690 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.091+0000 7fc39bfff640 10 monclient(hunting): dump: 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:58.079 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1380349390 0 0) 0x7fc3a0004eb0 con 0x7fc3ac030690 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 >> v1:172.21.15.157:6790/0 conn(0x7fc3ac0c98a0 legacy=0x7fc3ac15cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 >> v1:172.21.15.195:6789/0 conn(0x7fc3ac038a10 legacy=0x7fc3ac0c22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 1 monclient: found mon.a 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:18:58.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac0c2eb0 con 0x7fc3ac030690 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 10 monclient: _finish_auth 0 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc3b9838640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc3b9838640 10 monclient: _renew_subs 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc3b9838640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ac0c29b0 con 0x7fc3ac030690 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc3b9838640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc3b9838640 10 monclient: _renew_subs 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc3b9838640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ac0c2d00 con 0x7fc3ac030690 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc3b9838640 1 librados: init done 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3a4003940 con 0x7fc3ac030690 2024-03-29T11:18:58.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc3a4003c50 con 0x7fc3ac030690 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc3b17fa640 4 set_mon_vals no callback set 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.092+0000 7fc39bfff640 10 monclient: dump: 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:18:58.082 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:18:58.083 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:18:58.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.093+0000 7fc3b9838640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.094+0000 7fc3b9838640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.094+0000 7fc3b9838640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:18:58.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.094+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc394005080 con 0x7fc3ac030690 2024-03-29T11:18:58.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.095+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fc3a4019c20 con 0x7fc3ac030690 2024-03-29T11:18:58.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.096+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fc3a40838e0 con 0x7fc3ac030690 2024-03-29T11:18:58.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.096+0000 7fc39bfff640 10 client.4488.objecter ms_dispatch 0x7fc3b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:58.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.096+0000 7fc3b27fc640 10 monclient: get_auth_request con 0x7fc3a006ddc0 auth_method 0 2024-03-29T11:18:58.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.096+0000 7fc39bfff640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:58.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.096+0000 7fc39bfff640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:58.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.096+0000 7fc39bfff640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T11:18:58.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.096+0000 7fc39bfff640 10 client.4488.objecter ms_handle_connect 0x7fc3a006ddc0 2024-03-29T11:18:58.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.105+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3a404f220 con 0x7fc3ac030690 2024-03-29T11:18:58.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.106+0000 7fc39bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:58.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.106+0000 7fc39bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:58.494 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.516+0000 7fc3b9838640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:58.494 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.517+0000 7fc3b9838640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:58.494 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.517+0000 7fc3b9838640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:18:58.494 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.517+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc394005330 con 0x7fc3ac030690 2024-03-29T11:18:58.496 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.519+0000 7fc39bfff640 1 -- 172.21.15.157:0/1166464885 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fc3a4051150 con 0x7fc3ac030690 2024-03-29T11:18:58.496 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.519+0000 7fc39bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:58.496 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.519+0000 7fc39bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:58.497 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:18:58.497 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:58.500 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.523+0000 7fc3b9838640 10 librados: watch_flush enter 2024-03-29T11:18:58.500 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.523+0000 7fc3b9838640 10 librados: watch_flush exit 2024-03-29T11:18:58.500 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.523+0000 7fc3b9838640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T11:18:58.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.523+0000 7fc3b9838640 10 client.4488.objecter successfully canceled tick 2024-03-29T11:18:58.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.523+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 >> v1:172.21.15.157:6816/1747058247 conn(0x7fc3a006ddc0 legacy=0x7fc3a0070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.523+0000 7fc3b9838640 10 monclient: shutdown 2024-03-29T11:18:58.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.523+0000 7fc3b9838640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:58.501 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.523+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 >> v1:172.21.15.157:6789/0 conn(0x7fc3ac030690 legacy=0x7fc3ac0ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:58.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.524+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 shutdown_connections 2024-03-29T11:18:58.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.524+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 >> 172.21.15.157:0/1166464885 conn(0x7fc3ac005c30 msgr2=0x7fc3ac0340c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:58.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.524+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 shutdown_connections 2024-03-29T11:18:58.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.524+0000 7fc3b9838640 1 -- 172.21.15.157:0/1166464885 wait complete. 2024-03-29T11:18:58.502 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:18:58.524+0000 7fc3b9838640 1 librados: shutdown 2024-03-29T11:18:58.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:05.325 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:05.482 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.503+0000 7f2ada655640 10 monclient: get_monmap_and_config 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.503+0000 7f2ada655640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.503+0000 7f2ada655640 10 monclient: monmap: 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:19:05.505003+0000 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:19:05.505003+0000 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:19:05.483 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:05.484 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.505+0000 7f2ada655640 1 Processor -- start 2024-03-29T11:19:05.484 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.505+0000 7f2ada655640 1 -- start start 2024-03-29T11:19:05.484 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.505+0000 7f2ada655640 10 monclient: init 2024-03-29T11:19:05.485 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.506+0000 7f2ada655640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.485 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.506+0000 7f2ada655640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:05.487 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.508+0000 7f2ada655640 10 monclient(hunting): picked mon.noname-b con 0x7f2ad40ad6c0 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:05.487 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.508+0000 7f2ada655640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad4142db0 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:05.487 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.509+0000 7f2ada655640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad414ba40 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.509+0000 7f2ada655640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad40adaa0 con 0x7f2ad414ba40 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.509+0000 7f2ada655640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad40aab80 con 0x7f2ad4142db0 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.509+0000 7f2ada655640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad41425f0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.509+0000 7f2ada655640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.509+0000 7f2ada655640 10 monclient(hunting): authenticate will time out at 1357.705947s 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.509+0000 7f2ad3fff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f2ad40ad6c0 0x7f2ad41456b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:48785/0 (socket says 172.21.15.157:48785) 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.509+0000 7f2ad3fff640 1 -- 172.21.15.157:0/1365521684 learned_addr learned my addr 172.21.15.157:0/1365521684 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.510+0000 7f2ad2ffd640 1 -- 172.21.15.157:0/1365521684 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470048266 0 0) 0x7f2ad41425f0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.510+0000 7f2ad2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.510+0000 7f2ad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.488 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.510+0000 7f2ad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.489 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.510+0000 7f2ad2ffd640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:19:05.489 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.510+0000 7f2ad2ffd640 1 -- 172.21.15.157:0/1365521684 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2abc003520 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 1 -- 172.21.15.157:0/1365521684 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43248436 0 0) 0x7f2abc003520 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 1 -- 172.21.15.157:0/1365521684 >> v1:172.21.15.195:6789/0 conn(0x7f2ad4142db0 legacy=0x7f2ad40ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 1 -- 172.21.15.157:0/1365521684 >> v1:172.21.15.157:6789/0 conn(0x7f2ad414ba40 legacy=0x7f2ad414de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:05.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:19:05.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:19:05.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 1 -- 172.21.15.157:0/1365521684 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad40aae60 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.491 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 1 -- 172.21.15.157:0/1365521684 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2ac4001690 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.513+0000 7f2ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.514+0000 7f2ad2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.514+0000 7f2ad2ffd640 10 monclient: dump: 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.492 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:05.493 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:05.493 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:05.493 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:05.493 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:05.493 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.514+0000 7f2ada655640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:19:05.493 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.514+0000 7f2ada655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.493 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.516+0000 7f2ad2ffd640 1 -- 172.21.15.157:0/1365521684 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ac4001860 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.493 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.516+0000 7f2ad2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.516+0000 7f2ad2ffd640 1 -- 172.21.15.157:0/1365521684 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2ac4001cb0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.516+0000 7f2ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.516+0000 7f2ad2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.516+0000 7f2ad2ffd640 10 monclient: dump: 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:05.494 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:05.495 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:05.495 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:05.495 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:05.495 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.518+0000 7f2ada655640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:05.495 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.518+0000 7f2ada655640 4 set_mon_vals no callback set 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.519+0000 7f2ada655640 10 monclient: shutdown 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.519+0000 7f2ada655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.519+0000 7f2ada655640 1 -- 172.21.15.157:0/1365521684 >> v1:172.21.15.157:6790/0 conn(0x7f2ad40ad6c0 legacy=0x7f2ad41456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.519+0000 7f2ada655640 1 -- 172.21.15.157:0/1365521684 shutdown_connections 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.519+0000 7f2ada655640 1 -- 172.21.15.157:0/1365521684 >> 172.21.15.157:0/1365521684 conn(0x7f2ad40b35e0 msgr2=0x7f2ad40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.519+0000 7f2ada655640 1 -- 172.21.15.157:0/1365521684 shutdown_connections 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.519+0000 7f2ada655640 1 -- 172.21.15.157:0/1365521684 wait complete. 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.520+0000 7f2ada655640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.520+0000 7f2ada655640 10 monclient: monmap: 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:05.498 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:05.499 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:19:05.499 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:19:05.499 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.499 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:05.499 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:05.499 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:19:05.499 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:19:05.499 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:05.499 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.521+0000 7f2ada655640 1 librados: starting msgr at 2024-03-29T11:19:05.499 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.521+0000 7f2ada655640 1 librados: starting objecter 2024-03-29T11:19:05.500 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.521+0000 7f2ada655640 1 Processor -- start 2024-03-29T11:19:05.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.521+0000 7f2ada655640 1 -- start start 2024-03-29T11:19:05.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.521+0000 7f2ada655640 1 librados: setting wanted keys 2024-03-29T11:19:05.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.521+0000 7f2ada655640 1 librados: calling monclient init 2024-03-29T11:19:05.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.521+0000 7f2ada655640 10 monclient: init 2024-03-29T11:19:05.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.523+0000 7f2ada655640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.523+0000 7f2ada655640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:05.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.526+0000 7f2ada655640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad40ad6c0 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:05.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.526+0000 7f2ada655640 10 monclient(hunting): picked mon.noname-b con 0x7f2ad4142db0 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:05.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.526+0000 7f2ada655640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad414ba40 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:05.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.526+0000 7f2ada655640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad41daed0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.526+0000 7f2ada655640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad41db3f0 con 0x7f2ad4142db0 2024-03-29T11:19:05.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.526+0000 7f2ada655640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad414b610 con 0x7f2ad414ba40 2024-03-29T11:19:05.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.526+0000 7f2ada655640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.526+0000 7f2ada655640 10 monclient(hunting): authenticate will time out at 1357.723028s 2024-03-29T11:19:05.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.528+0000 7f2ad3fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f2ad40ad6c0 0x7f2ad4072030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:57213/0 (socket says 172.21.15.157:57213) 2024-03-29T11:19:05.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.528+0000 7f2ad3fff640 1 -- 172.21.15.157:0/911928589 learned_addr learned my addr 172.21.15.157:0/911928589 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:05.508 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.529+0000 7f2ad0ff9640 10 client.?.objecter ms_handle_connect 0x7f2ad40ad6c0 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.529+0000 7f2ad0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.529+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250041146 0 0) 0x7f2ad41daed0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.529+0000 7f2ad0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.529+0000 7f2ad0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.529+0000 7f2ad0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.529+0000 7f2ad0ff9640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.529+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac0004eb0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.529+0000 7f2ad0ff9640 10 client.?.objecter ms_handle_connect 0x7f2ad414ba40 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.529+0000 7f2ad0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 client.?.objecter ms_handle_connect 0x7f2ad4142db0 2024-03-29T11:19:05.509 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412158892 0 0) 0x7f2ad414b610 con 0x7f2ad414ba40 2024-03-29T11:19:05.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:19:05.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad41daed0 con 0x7f2ad414ba40 2024-03-29T11:19:05.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2936065581 0 0) 0x7f2ac0004eb0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad414b610 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2ac4001b70 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:05.511 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:05.511 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:05.511 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:05.511 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:05.511 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:05.511 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.511 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1838700735 0 0) 0x7f2ad41daed0 con 0x7f2ad414ba40 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ac0004eb0 con 0x7f2ad414ba40 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2acc001a70 con 0x7f2ad414ba40 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:05.512 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2312616645 0 0) 0x7f2ad414b610 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 >> v1:172.21.15.157:6790/0 conn(0x7f2ad414ba40 legacy=0x7f2ad40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.513 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 >> v1:172.21.15.195:6789/0 conn(0x7f2ad4142db0 legacy=0x7f2ad41da5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 monclient: found mon.a 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad414b7a0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ad0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ada655640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ada655640 10 monclient: _renew_subs 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ada655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:19:05.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.530+0000 7f2ada655640 1 -- 172.21.15.157:0/911928589 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad411b1c0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.531+0000 7f2ada655640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.531+0000 7f2ada655640 10 monclient: _renew_subs 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.531+0000 7f2ada655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.531+0000 7f2ada655640 1 -- 172.21.15.157:0/911928589 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad411b6f0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.531+0000 7f2ada655640 1 librados: init done 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.532+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ac4004bf0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.532+0000 7f2ad0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.532+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f2ac4001cd0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.532+0000 7f2ad0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.532+0000 7f2ad0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.532+0000 7f2ad0ff9640 10 monclient: dump: 2024-03-29T11:19:05.515 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:05.516 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:05.516 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:05.516 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:05.516 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.516 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:05.516 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:05.516 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:05.516 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:05.516 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:05.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.532+0000 7f2ad27fc640 4 set_mon_vals no callback set 2024-03-29T11:19:05.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.534+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f2ac4004fc0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.534+0000 7f2aba7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.534+0000 7f2aba7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.534+0000 7f2aba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:19:05.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.534+0000 7f2aba7fc640 1 -- 172.21.15.157:0/911928589 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab4005080 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.517 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.535+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f2ac4084bb0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.535+0000 7f2ad0ff9640 10 client.4491.objecter ms_dispatch 0x7f2ad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:05.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.535+0000 7f2ad0ff9640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:05.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.535+0000 7f2ad0ff9640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:05.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.535+0000 7f2ad0ff9640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-29T11:19:05.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.535+0000 7f2ad37fe640 10 monclient: get_auth_request con 0x7f2ac0076690 auth_method 0 2024-03-29T11:19:05.518 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.536+0000 7f2ad0ff9640 10 client.4491.objecter ms_handle_connect 0x7f2ac0076690 2024-03-29T11:19:05.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.544+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ac4050490 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.544+0000 7f2ad0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.544+0000 7f2ad0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:05.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.908+0000 7f2aba7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:05.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.908+0000 7f2aba7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:05.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.908+0000 7f2aba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:19:05.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.908+0000 7f2aba7fc640 1 -- 172.21.15.157:0/911928589 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ab4005330 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.909+0000 7f2ad0ff9640 1 -- 172.21.15.157:0/911928589 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f2ac40523c0 con 0x7f2ad40ad6c0 2024-03-29T11:19:05.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.909+0000 7f2ad0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:05.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.909+0000 7f2ad0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:05.887 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:19:05.887 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:05.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.913+0000 7f2aba7fc640 10 librados: watch_flush enter 2024-03-29T11:19:05.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.913+0000 7f2aba7fc640 10 librados: watch_flush exit 2024-03-29T11:19:05.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.913+0000 7f2aba7fc640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-29T11:19:05.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.913+0000 7f2aba7fc640 10 client.4491.objecter successfully canceled tick 2024-03-29T11:19:05.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.914+0000 7f2aba7fc640 1 -- 172.21.15.157:0/911928589 >> v1:172.21.15.157:6816/1747058247 conn(0x7f2ac0076690 legacy=0x7f2ac0078b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.914+0000 7f2aba7fc640 10 monclient: shutdown 2024-03-29T11:19:05.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.914+0000 7f2aba7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.914+0000 7f2aba7fc640 1 -- 172.21.15.157:0/911928589 >> v1:172.21.15.157:6789/0 conn(0x7f2ad40ad6c0 legacy=0x7f2ad4072030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.914+0000 7f2aba7fc640 1 -- 172.21.15.157:0/911928589 shutdown_connections 2024-03-29T11:19:05.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.914+0000 7f2aba7fc640 1 -- 172.21.15.157:0/911928589 >> 172.21.15.157:0/911928589 conn(0x7f2ad40b35e0 msgr2=0x7f2ad413efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.914+0000 7f2aba7fc640 1 -- 172.21.15.157:0/911928589 shutdown_connections 2024-03-29T11:19:05.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.914+0000 7f2aba7fc640 1 -- 172.21.15.157:0/911928589 wait complete. 2024-03-29T11:19:05.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:05.915+0000 7f2aba7fc640 1 librados: shutdown 2024-03-29T11:19:05.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:12.714 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:12.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.887+0000 7fe3db4f1640 10 monclient: get_monmap_and_config 2024-03-29T11:19:12.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.887+0000 7fe3db4f1640 10 monclient: build_initial_monmap 2024-03-29T11:19:12.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.887+0000 7fe3db4f1640 10 monclient: monmap: 2024-03-29T11:19:12.864 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:12.864 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:12.864 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:19:12.888177+0000 2024-03-29T11:19:12.864 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:19:12.888177+0000 2024-03-29T11:19:12.865 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:12.865 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:12.865 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:12.865 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:19:12.865 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:19:12.865 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:12.865 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.888+0000 7fe3db4f1640 1 Processor -- start 2024-03-29T11:19:12.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 1 -- start start 2024-03-29T11:19:12.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 10 monclient: init 2024-03-29T11:19:12.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:12.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:12.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 10 monclient(hunting): picked mon.noname-c con 0x7fe3d406d9d0 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:12.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 10 monclient(hunting): picked mon.noname-a con 0x7fe3d413e820 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:12.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 10 monclient(hunting): picked mon.noname-b con 0x7fe3d40af1d0 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:12.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d40aed10 con 0x7fe3d413e820 2024-03-29T11:19:12.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d40afd50 con 0x7fe3d406d9d0 2024-03-29T11:19:12.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d40ab490 con 0x7fe3d40af1d0 2024-03-29T11:19:12.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:12.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.889+0000 7fe3db4f1640 10 monclient(hunting): authenticate will time out at 1365.086783s 2024-03-29T11:19:12.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.893+0000 7fe3d8a8c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe3d413e820 0x7fe3d413ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:40809/0 (socket says 172.21.15.157:40809) 2024-03-29T11:19:12.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.893+0000 7fe3d8a8c640 1 -- 172.21.15.157:0/1039811513 learned_addr learned my addr 172.21.15.157:0/1039811513 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:12.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438860739 0 0) 0x7fe3d40aed10 con 0x7fe3d413e820 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c0003520 con 0x7fe3d413e820 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751548413 0 0) 0x7fe3d40ab490 con 0x7fe3d40af1d0 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d40aed10 con 0x7fe3d40af1d0 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3968419017 0 0) 0x7fe3d40aed10 con 0x7fe3d40af1d0 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 >> v1:172.21.15.195:6789/0 conn(0x7fe3d406d9d0 legacy=0x7fe3d40b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 >> v1:172.21.15.157:6789/0 conn(0x7fe3d413e820 legacy=0x7fe3d413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:12.873 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.895+0000 7fe3cbfff640 1 monclient: found mon.noname-b 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d40ab770 con 0x7fe3d40af1d0 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fe3d0001670 con 0x7fe3d40af1d0 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 10 monclient: dump: 2024-03-29T11:19:12.874 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3db4f1640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:19:12.875 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3db4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3d00017d0 con 0x7fe3d40af1d0 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 1 -- 172.21.15.157:0/1039811513 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fe3d0001ba0 con 0x7fe3d40af1d0 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3cbfff640 10 monclient: dump: 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:12.876 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3db4f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.896+0000 7fe3db4f1640 4 set_mon_vals no callback set 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.897+0000 7fe3db4f1640 10 monclient: shutdown 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.897+0000 7fe3db4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.897+0000 7fe3db4f1640 1 -- 172.21.15.157:0/1039811513 >> v1:172.21.15.157:6790/0 conn(0x7fe3d40af1d0 legacy=0x7fe3d40af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.898+0000 7fe3db4f1640 1 -- 172.21.15.157:0/1039811513 shutdown_connections 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.898+0000 7fe3db4f1640 1 -- 172.21.15.157:0/1039811513 >> 172.21.15.157:0/1039811513 conn(0x7fe3d406b660 msgr2=0x7fe3d406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:12.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.898+0000 7fe3db4f1640 1 -- 172.21.15.157:0/1039811513 shutdown_connections 2024-03-29T11:19:12.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.898+0000 7fe3db4f1640 1 -- 172.21.15.157:0/1039811513 wait complete. 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.899+0000 7fe3db4f1640 10 monclient: build_initial_monmap 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.899+0000 7fe3db4f1640 10 monclient: monmap: 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.899+0000 7fe3db4f1640 1 librados: starting msgr at 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.899+0000 7fe3db4f1640 1 librados: starting objecter 2024-03-29T11:19:12.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.899+0000 7fe3db4f1640 1 Processor -- start 2024-03-29T11:19:12.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.900+0000 7fe3db4f1640 1 -- start start 2024-03-29T11:19:12.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.900+0000 7fe3db4f1640 1 librados: setting wanted keys 2024-03-29T11:19:12.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.900+0000 7fe3db4f1640 1 librados: calling monclient init 2024-03-29T11:19:12.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.900+0000 7fe3db4f1640 10 monclient: init 2024-03-29T11:19:12.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.900+0000 7fe3db4f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:12.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.900+0000 7fe3db4f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:12.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.900+0000 7fe3db4f1640 10 monclient(hunting): picked mon.noname-b con 0x7fe3d406d9d0 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:12.880 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.904+0000 7fe3db4f1640 10 monclient(hunting): picked mon.noname-c con 0x7fe3d40af1d0 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:12.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.904+0000 7fe3db4f1640 10 monclient(hunting): picked mon.noname-a con 0x7fe3d413e820 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:12.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.906+0000 7fe3db4f1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d41dbdd0 con 0x7fe3d413e820 2024-03-29T11:19:12.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.906+0000 7fe3db4f1640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d406e6b0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.906+0000 7fe3db4f1640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d406fb90 con 0x7fe3d40af1d0 2024-03-29T11:19:12.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.906+0000 7fe3db4f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:12.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.906+0000 7fe3db4f1640 10 monclient(hunting): authenticate will time out at 1365.103779s 2024-03-29T11:19:12.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.907+0000 7fe3d928d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe3d406d9d0 0x7fe3d41e7500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:36775/0 (socket says 172.21.15.157:36775) 2024-03-29T11:19:12.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.907+0000 7fe3d928d640 1 -- 172.21.15.157:0/4043079300 learned_addr learned my addr 172.21.15.157:0/4043079300 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:12.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.907+0000 7fe3c9ffb640 10 client.?.objecter ms_handle_connect 0x7fe3d406d9d0 2024-03-29T11:19:12.884 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.907+0000 7fe3c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:12.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.907+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559139020 0 0) 0x7fe3d406e6b0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.907+0000 7fe3c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:12.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.907+0000 7fe3c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:12.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.907+0000 7fe3c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:12.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.907+0000 7fe3c9ffb640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:19:12.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.907+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b8004eb0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 226726874 0 0) 0x7fe3b8004eb0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3d406e6b0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fe3c4001990 con 0x7fe3d406d9d0 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:12.887 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2223950148 0 0) 0x7fe3d406e6b0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 >> v1:172.21.15.157:6790/0 conn(0x7fe3d40af1d0 legacy=0x7fe3d41dafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 >> v1:172.21.15.157:6789/0 conn(0x7fe3d413e820 legacy=0x7fe3d41db6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 1 monclient: found mon.b 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d406e190 con 0x7fe3d406d9d0 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3db4f1640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:19:12.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3db4f1640 10 monclient: _renew_subs 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3db4f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.908+0000 7fe3db4f1640 1 -- 172.21.15.157:0/4043079300 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d406ddb0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.909+0000 7fe3db4f1640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.909+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3c4001af0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.909+0000 7fe3c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.909+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fe3c4003a60 con 0x7fe3d406d9d0 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.909+0000 7fe3c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.909+0000 7fe3c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.909+0000 7fe3c9ffb640 10 monclient: dump: 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:12.889 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:12.890 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:12.890 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:12.890 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:12.890 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:12.890 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.909+0000 7fe3cb7fe640 4 set_mon_vals no callback set 2024-03-29T11:19:12.890 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.910+0000 7fe3db4f1640 10 monclient: _renew_subs 2024-03-29T11:19:12.890 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.910+0000 7fe3db4f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:12.890 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.910+0000 7fe3db4f1640 1 -- 172.21.15.157:0/4043079300 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d406eb10 con 0x7fe3d406d9d0 2024-03-29T11:19:12.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.910+0000 7fe3db4f1640 1 librados: init done 2024-03-29T11:19:12.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fe3c4003bc0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fe3c4084b40 con 0x7fe3d406d9d0 2024-03-29T11:19:12.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3c9ffb640 10 client.4492.objecter ms_dispatch 0x7fe3d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:12.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3c9ffb640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:12.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3c9ffb640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:12.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3c9ffb640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-29T11:19:12.891 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3d8a8c640 10 monclient: get_auth_request con 0x7fe3b806dd20 auth_method 0 2024-03-29T11:19:12.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3db4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:12.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3db4f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:12.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3db4f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:12.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.913+0000 7fe3db4f1640 1 -- 172.21.15.157:0/4043079300 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d406e3d0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.892 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.914+0000 7fe3c9ffb640 10 client.4492.objecter ms_handle_connect 0x7fe3b806dd20 2024-03-29T11:19:12.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.922+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3d406e3d0 con 0x7fe3d406d9d0 2024-03-29T11:19:12.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.922+0000 7fe3c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:12.899 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:12.922+0000 7fe3c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:13.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.243+0000 7fe3db4f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:13.220 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.243+0000 7fe3db4f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:13.221 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.243+0000 7fe3db4f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:13.221 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.243+0000 7fe3db4f1640 1 -- 172.21.15.157:0/4043079300 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3d406e3d0 con 0x7fe3d406d9d0 2024-03-29T11:19:13.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.248+0000 7fe3c9ffb640 1 -- 172.21.15.157:0/4043079300 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fe3d406e3d0 con 0x7fe3d406d9d0 2024-03-29T11:19:13.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.248+0000 7fe3c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:13.225 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.248+0000 7fe3c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:13.226 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:19:13.226 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:13.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.251+0000 7fe3db4f1640 10 librados: watch_flush enter 2024-03-29T11:19:13.230 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.252+0000 7fe3db4f1640 10 librados: watch_flush exit 2024-03-29T11:19:13.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.252+0000 7fe3db4f1640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-29T11:19:13.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.252+0000 7fe3db4f1640 10 client.4492.objecter successfully canceled tick 2024-03-29T11:19:13.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.254+0000 7fe3db4f1640 1 -- 172.21.15.157:0/4043079300 >> v1:172.21.15.157:6816/1747058247 conn(0x7fe3b806dd20 legacy=0x7fe3b80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:13.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.254+0000 7fe3db4f1640 10 monclient: shutdown 2024-03-29T11:19:13.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.254+0000 7fe3db4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:13.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.254+0000 7fe3db4f1640 1 -- 172.21.15.157:0/4043079300 >> v1:172.21.15.195:6789/0 conn(0x7fe3d406d9d0 legacy=0x7fe3d41e7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:13.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.257+0000 7fe3db4f1640 1 -- 172.21.15.157:0/4043079300 shutdown_connections 2024-03-29T11:19:13.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.257+0000 7fe3db4f1640 1 -- 172.21.15.157:0/4043079300 >> 172.21.15.157:0/4043079300 conn(0x7fe3d406b660 msgr2=0x7fe3d413c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:13.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.259+0000 7fe3db4f1640 1 -- 172.21.15.157:0/4043079300 shutdown_connections 2024-03-29T11:19:13.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.259+0000 7fe3db4f1640 1 -- 172.21.15.157:0/4043079300 wait complete. 2024-03-29T11:19:13.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:13.259+0000 7fe3db4f1640 1 librados: shutdown 2024-03-29T11:19:13.259 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:20.065 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.250+0000 7fb0585a8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.250+0000 7fb0585a8640 10 monclient: build_initial_monmap 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.250+0000 7fb0585a8640 10 monclient: monmap: 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:19:20.250850+0000 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:19:20.250850+0000 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:19:20.227 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:19:20.228 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:20.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.251+0000 7fb0585a8640 1 Processor -- start 2024-03-29T11:19:20.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.251+0000 7fb0585a8640 1 -- start start 2024-03-29T11:19:20.228 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.251+0000 7fb0585a8640 10 monclient: init 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.252+0000 7fb0585a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.252+0000 7fb0585a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.252+0000 7fb0585a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb0500b0770 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.252+0000 7fb0585a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb05014a700 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.252+0000 7fb0585a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb0500ac5a0 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.252+0000 7fb0585a8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0500aba70 con 0x7fb0500b0770 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.252+0000 7fb0585a8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0500aecb0 con 0x7fb05014a700 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.252+0000 7fb0585a8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0500af150 con 0x7fb0500ac5a0 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.252+0000 7fb0585a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.252+0000 7fb0585a8640 10 monclient(hunting): authenticate will time out at 1372.448842s 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb056b45640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fb0500ac5a0 0x7fb050151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:35816/0 (socket says 172.21.15.157:35816) 2024-03-29T11:19:20.229 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb056b45640 1 -- 172.21.15.157:0/1987894447 learned_addr learned my addr 172.21.15.157:0/1987894447 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:20.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279981700 0 0) 0x7fb0500af150 con 0x7fb0500ac5a0 2024-03-29T11:19:20.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 10 monclient(hunting): _negotiate 2024-03-29T11:19:20.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:20.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:20.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:19:20.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03c003520 con 0x7fb0500ac5a0 2024-03-29T11:19:20.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440691127 0 0) 0x7fb0500aba70 con 0x7fb0500b0770 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 10 monclient(hunting): _negotiate 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.253+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03c003840 con 0x7fb0500b0770 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2028258742 0 0) 0x7fb03c003520 con 0x7fb0500ac5a0 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 >> v1:172.21.15.195:6789/0 conn(0x7fb05014a700 legacy=0x7fb0500abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 >> v1:172.21.15.157:6789/0 conn(0x7fb0500b0770 legacy=0x7fb050149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: _finish_hunting 0 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 1 monclient: found mon.noname-b 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:19:20.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0500ac9d0 con 0x7fb0500ac5a0 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: _finish_auth 0 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: _check_auth_tickets 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb0480019f0 con 0x7fb0500ac5a0 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: dump: 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:20.233 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb0585a8640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb0585a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb048001b50 con 0x7fb0500ac5a0 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 1 -- 172.21.15.157:0/1987894447 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb048001f60 con 0x7fb0500ac5a0 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb055342640 10 monclient: dump: 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:20.234 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb0585a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.254+0000 7fb0585a8640 4 set_mon_vals no callback set 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.255+0000 7fb0585a8640 10 monclient: shutdown 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.255+0000 7fb0585a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.255+0000 7fb0585a8640 1 -- 172.21.15.157:0/1987894447 >> v1:172.21.15.157:6790/0 conn(0x7fb0500ac5a0 legacy=0x7fb050151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.255+0000 7fb0585a8640 1 -- 172.21.15.157:0/1987894447 shutdown_connections 2024-03-29T11:19:20.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.255+0000 7fb0585a8640 1 -- 172.21.15.157:0/1987894447 >> 172.21.15.157:0/1987894447 conn(0x7fb05006b660 msgr2=0x7fb05006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.255+0000 7fb0585a8640 1 -- 172.21.15.157:0/1987894447 shutdown_connections 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.256+0000 7fb0585a8640 1 -- 172.21.15.157:0/1987894447 wait complete. 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.256+0000 7fb0585a8640 10 monclient: build_initial_monmap 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.256+0000 7fb0585a8640 10 monclient: monmap: 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:20.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.257+0000 7fb0585a8640 1 librados: starting msgr at 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.257+0000 7fb0585a8640 1 librados: starting objecter 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.257+0000 7fb0585a8640 1 Processor -- start 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.257+0000 7fb0585a8640 1 -- start start 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.257+0000 7fb0585a8640 1 librados: setting wanted keys 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.257+0000 7fb0585a8640 1 librados: calling monclient init 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.257+0000 7fb0585a8640 10 monclient: init 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb0585a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb0585a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb0585a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb0500ac5a0 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb0585a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb0500b0770 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb055b43640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb0500b0770 0x7fb0501d1f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:38572/0 (socket says 172.21.15.157:38572) 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb055b43640 1 -- 172.21.15.157:0/3873358882 learned_addr learned my addr 172.21.15.157:0/3873358882 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb0585a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb05014a700 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:20.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0501244a0 con 0x7fb0500b0770 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb050124780 con 0x7fb05014a700 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0501d3130 con 0x7fb0500ac5a0 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb0585a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb0585a8640 10 monclient(hunting): authenticate will time out at 1372.454652s 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb046ffd640 10 client.?.objecter ms_handle_connect 0x7fb0500b0770 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.258+0000 7fb046ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 client.?.objecter ms_handle_connect 0x7fb05014a700 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896393674 0 0) 0x7fb0501244a0 con 0x7fb0500b0770 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb034004eb0 con 0x7fb0500b0770 2024-03-29T11:19:20.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 client.?.objecter ms_handle_connect 0x7fb0500ac5a0 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320754727 0 0) 0x7fb050124780 con 0x7fb05014a700 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0501244a0 con 0x7fb05014a700 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2354896911 0 0) 0x7fb034004eb0 con 0x7fb0500b0770 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb050124780 con 0x7fb0500b0770 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb040003fe0 con 0x7fb0500b0770 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:20.244 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.259+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 581526440 0 0) 0x7fb0501244a0 con 0x7fb05014a700 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb034004eb0 con 0x7fb05014a700 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb048001e60 con 0x7fb05014a700 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:20.245 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487188787 0 0) 0x7fb0501d3130 con 0x7fb0500ac5a0 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0501244a0 con 0x7fb0500ac5a0 2024-03-29T11:19:20.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 420919930 0 0) 0x7fb034004eb0 con 0x7fb05014a700 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 >> v1:172.21.15.157:6790/0 conn(0x7fb0500ac5a0 legacy=0x7fb0501df050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 >> v1:172.21.15.157:6789/0 conn(0x7fb0500b0770 legacy=0x7fb0501d1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 monclient: found mon.b 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0501d2da0 con 0x7fb05014a700 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb0585a8640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb0585a8640 10 monclient: _renew_subs 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb0585a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb050144ef0 con 0x7fb05014a700 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb0585a8640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:20.247 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb0585a8640 10 monclient: _renew_subs 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb0585a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0501cf860 con 0x7fb05014a700 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb0585a8640 1 librados: init done 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb048001590 con 0x7fb05014a700 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fb048004fb0 con 0x7fb05014a700 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb046ffd640 10 monclient: dump: 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:20.248 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.260+0000 7fb054b41640 4 set_mon_vals no callback set 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.262+0000 7fb0585a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.262+0000 7fb0585a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.262+0000 7fb0585a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.262+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb038005080 con 0x7fb05014a700 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.262+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fb04801af80 con 0x7fb05014a700 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.263+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fb048084e50 con 0x7fb05014a700 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.263+0000 7fb046ffd640 10 client.4495.objecter ms_dispatch 0x7fb050000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.263+0000 7fb046ffd640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.263+0000 7fb046ffd640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.263+0000 7fb046ffd640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.263+0000 7fb056344640 10 monclient: get_auth_request con 0x7fb03406df30 auth_method 0 2024-03-29T11:19:20.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.264+0000 7fb046ffd640 10 client.4495.objecter ms_handle_connect 0x7fb03406df30 2024-03-29T11:19:20.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.271+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb048050730 con 0x7fb05014a700 2024-03-29T11:19:20.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.271+0000 7fb046ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:20.250 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.271+0000 7fb046ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:20.601 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.623+0000 7fb0585a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:20.601 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.623+0000 7fb0585a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:20.601 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.623+0000 7fb0585a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:20.601 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.623+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb038005330 con 0x7fb05014a700 2024-03-29T11:19:20.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.624+0000 7fb046ffd640 1 -- 172.21.15.157:0/3873358882 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fb048052660 con 0x7fb05014a700 2024-03-29T11:19:20.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.624+0000 7fb046ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:20.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.624+0000 7fb046ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:20.602 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:19:20.602 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:20.605 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.628+0000 7fb0585a8640 10 librados: watch_flush enter 2024-03-29T11:19:20.605 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.628+0000 7fb0585a8640 10 librados: watch_flush exit 2024-03-29T11:19:20.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.628+0000 7fb0585a8640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T11:19:20.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.628+0000 7fb0585a8640 10 client.4495.objecter successfully canceled tick 2024-03-29T11:19:20.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.628+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 >> v1:172.21.15.157:6816/1747058247 conn(0x7fb03406df30 legacy=0x7fb0340703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:20.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.628+0000 7fb0585a8640 10 monclient: shutdown 2024-03-29T11:19:20.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.628+0000 7fb0585a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:20.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.628+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 >> v1:172.21.15.195:6789/0 conn(0x7fb05014a700 legacy=0x7fb0501d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:20.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.629+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 shutdown_connections 2024-03-29T11:19:20.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.629+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 >> 172.21.15.157:0/3873358882 conn(0x7fb05006b660 msgr2=0x7fb0500b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:20.607 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.629+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 shutdown_connections 2024-03-29T11:19:20.607 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.629+0000 7fb0585a8640 1 -- 172.21.15.157:0/3873358882 wait complete. 2024-03-29T11:19:20.607 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:20.629+0000 7fb0585a8640 1 librados: shutdown 2024-03-29T11:19:20.622 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:27.429 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:27.562 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.585+0000 7fc51d455640 10 monclient: get_monmap_and_config 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.585+0000 7fc51d455640 10 monclient: build_initial_monmap 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.585+0000 7fc51d455640 10 monclient: monmap: 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:19:27.586825+0000 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:19:27.586825+0000 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:19:27.563 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:27.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.586+0000 7fc51d455640 1 Processor -- start 2024-03-29T11:19:27.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.586+0000 7fc51d455640 1 -- start start 2024-03-29T11:19:27.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.586+0000 7fc51d455640 10 monclient: init 2024-03-29T11:19:27.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.587+0000 7fc51d455640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:27.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.587+0000 7fc51d455640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:27.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.587+0000 7fc51d455640 10 monclient(hunting): picked mon.noname-a con 0x7fc5180b0860 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:27.564 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.587+0000 7fc51d455640 10 monclient(hunting): picked mon.noname-b con 0x7fc518072450 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:27.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc50e7fc640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fc518072450 0x7fc51813c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:52060/0 (socket says 172.21.15.157:52060) 2024-03-29T11:19:27.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc50e7fc640 1 -- 172.21.15.157:0/3046177628 learned_addr learned my addr 172.21.15.157:0/3046177628 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:27.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc51d455640 10 monclient(hunting): picked mon.noname-c con 0x7fc51813ee40 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:27.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc51d455640 1 -- 172.21.15.157:0/3046177628 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5180ad840 con 0x7fc5180b0860 2024-03-29T11:19:27.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc51d455640 1 -- 172.21.15.157:0/3046177628 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51813c830 con 0x7fc51813ee40 2024-03-29T11:19:27.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc51d455640 1 -- 172.21.15.157:0/3046177628 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51813c9c0 con 0x7fc518072450 2024-03-29T11:19:27.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc51d455640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:27.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc51d455640 10 monclient(hunting): authenticate will time out at 1379.785562s 2024-03-29T11:19:27.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc5167fc640 1 -- 172.21.15.157:0/3046177628 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564296598 0 0) 0x7fc51813c830 con 0x7fc51813ee40 2024-03-29T11:19:27.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc5167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc5167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc5167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc5167fc640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:19:27.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.588+0000 7fc5167fc640 1 -- 172.21.15.157:0/3046177628 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc504003520 con 0x7fc51813ee40 2024-03-29T11:19:27.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.589+0000 7fc5167fc640 1 -- 172.21.15.157:0/3046177628 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 78236400 0 0) 0x7fc504003520 con 0x7fc51813ee40 2024-03-29T11:19:27.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.589+0000 7fc5167fc640 1 -- 172.21.15.157:0/3046177628 >> v1:172.21.15.157:6790/0 conn(0x7fc518072450 legacy=0x7fc51813c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.589+0000 7fc5167fc640 1 -- 172.21.15.157:0/3046177628 >> v1:172.21.15.157:6789/0 conn(0x7fc5180b0860 legacy=0x7fc5180acfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:27.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:27.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 1 monclient: found mon.noname-c 2024-03-29T11:19:27.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:27.568 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 1 -- 172.21.15.157:0/3046177628 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51813f970 con 0x7fc51813ee40 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 1 -- 172.21.15.157:0/3046177628 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc510001690 con 0x7fc51813ee40 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc5167fc640 10 monclient: dump: 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:27.569 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:27.570 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:27.570 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:27.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc51d455640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:19:27.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.590+0000 7fc51d455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:27.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.591+0000 7fc5167fc640 1 -- 172.21.15.157:0/3046177628 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc510001920 con 0x7fc51813ee40 2024-03-29T11:19:27.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.591+0000 7fc5167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:27.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.591+0000 7fc5167fc640 1 -- 172.21.15.157:0/3046177628 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc510001cf0 con 0x7fc51813ee40 2024-03-29T11:19:27.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.591+0000 7fc5167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.591+0000 7fc5167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.591+0000 7fc5167fc640 10 monclient: dump: 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.591+0000 7fc51d455640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.591+0000 7fc51d455640 4 set_mon_vals no callback set 2024-03-29T11:19:27.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.592+0000 7fc51d455640 10 monclient: shutdown 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.592+0000 7fc51d455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.592+0000 7fc51d455640 1 -- 172.21.15.157:0/3046177628 >> v1:172.21.15.195:6789/0 conn(0x7fc51813ee40 legacy=0x7fc51813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.593+0000 7fc51d455640 1 -- 172.21.15.157:0/3046177628 shutdown_connections 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.593+0000 7fc51d455640 1 -- 172.21.15.157:0/3046177628 >> 172.21.15.157:0/3046177628 conn(0x7fc51806b660 msgr2=0x7fc51806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.593+0000 7fc51d455640 1 -- 172.21.15.157:0/3046177628 shutdown_connections 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.593+0000 7fc51d455640 1 -- 172.21.15.157:0/3046177628 wait complete. 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.594+0000 7fc51d455640 10 monclient: build_initial_monmap 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.594+0000 7fc51d455640 10 monclient: monmap: 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:27.573 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:19:27.574 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:19:27.574 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:27.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.594+0000 7fc51d455640 1 librados: starting msgr at 2024-03-29T11:19:27.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.594+0000 7fc51d455640 1 librados: starting objecter 2024-03-29T11:19:27.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.594+0000 7fc51d455640 1 Processor -- start 2024-03-29T11:19:27.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.594+0000 7fc51d455640 1 -- start start 2024-03-29T11:19:27.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.594+0000 7fc51d455640 1 librados: setting wanted keys 2024-03-29T11:19:27.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.594+0000 7fc51d455640 1 librados: calling monclient init 2024-03-29T11:19:27.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.594+0000 7fc51d455640 10 monclient: init 2024-03-29T11:19:27.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.595+0000 7fc51d455640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:27.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.595+0000 7fc51d455640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.595+0000 7fc51d455640 10 monclient(hunting): picked mon.noname-b con 0x7fc518072450 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.596+0000 7fc516ffd640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc518072450 0x7fc5180b2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:38642/0 (socket says 172.21.15.157:38642) 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.596+0000 7fc51d455640 10 monclient(hunting): picked mon.noname-a con 0x7fc5180b0860 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.596+0000 7fc51d455640 10 monclient(hunting): picked mon.noname-c con 0x7fc51813ee40 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.596+0000 7fc51d455640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5180b2ef0 con 0x7fc5180b0860 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.596+0000 7fc51d455640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5180b39b0 con 0x7fc518072450 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.596+0000 7fc51d455640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5180b3c20 con 0x7fc51813ee40 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.596+0000 7fc51d455640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.596+0000 7fc51d455640 10 monclient(hunting): authenticate will time out at 1379.793581s 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.596+0000 7fc516ffd640 1 -- 172.21.15.157:0/3820383761 learned_addr learned my addr 172.21.15.157:0/3820383761 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.597+0000 7fc50ffff640 10 client.?.objecter ms_handle_connect 0x7fc51813ee40 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.597+0000 7fc50ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.597+0000 7fc50ffff640 10 client.?.objecter ms_handle_connect 0x7fc518072450 2024-03-29T11:19:27.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.597+0000 7fc50ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.597+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759986865 0 0) 0x7fc5180b3c20 con 0x7fc51813ee40 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.597+0000 7fc50ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.597+0000 7fc50ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.597+0000 7fc50ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.597+0000 7fc50ffff640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.597+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f8004eb0 con 0x7fc51813ee40 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385663377 0 0) 0x7fc5180b39b0 con 0x7fc518072450 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5180b3c20 con 0x7fc518072450 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3378339080 0 0) 0x7fc4f8004eb0 con 0x7fc51813ee40 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5180b39b0 con 0x7fc51813ee40 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc510001650 con 0x7fc51813ee40 2024-03-29T11:19:27.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): dump: 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1462848786 0 0) 0x7fc5180b3c20 con 0x7fc518072450 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4f8004eb0 con 0x7fc518072450 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc508001ca0 con 0x7fc518072450 2024-03-29T11:19:27.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 10 monclient(hunting): dump: 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2975399405 0 0) 0x7fc5180b39b0 con 0x7fc51813ee40 2024-03-29T11:19:27.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 >> v1:172.21.15.195:6789/0 conn(0x7fc518072450 legacy=0x7fc5180b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.598+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 >> v1:172.21.15.157:6789/0 conn(0x7fc5180b0860 legacy=0x7fc51812d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc50ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc50ffff640 1 monclient: found mon.c 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc50ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc50ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5180b3f00 con 0x7fc51813ee40 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc50ffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc50ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc50ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc51d455640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc51d455640 10 monclient: _renew_subs 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc51d455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc51d455640 1 -- 172.21.15.157:0/3820383761 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5180b2090 con 0x7fc51813ee40 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc51d455640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:27.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc51d455640 10 monclient: _renew_subs 2024-03-29T11:19:27.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc51d455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:19:27.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc51d455640 1 -- 172.21.15.157:0/3820383761 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5181e3000 con 0x7fc51813ee40 2024-03-29T11:19:27.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.599+0000 7fc51d455640 1 librados: init done 2024-03-29T11:19:27.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.600+0000 7fc51d455640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.600+0000 7fc51d455640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.600+0000 7fc51d455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.600+0000 7fc51d455640 1 -- 172.21.15.157:0/3820383761 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5180b0c40 con 0x7fc51813ee40 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.600+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc510001270 con 0x7fc51813ee40 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.600+0000 7fc50ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.600+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc510003a40 con 0x7fc51813ee40 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.600+0000 7fc50ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.600+0000 7fc50ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.600+0000 7fc50ffff640 10 monclient: dump: 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.585 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:27.586 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:27.586 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:27.586 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:27.586 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:27.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.601+0000 7fc515ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:27.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.602+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fc510019a10 con 0x7fc51813ee40 2024-03-29T11:19:27.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.602+0000 7fc516ffd640 10 monclient: get_auth_request con 0x7fc4f806de30 auth_method 0 2024-03-29T11:19:27.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.602+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fc510083900 con 0x7fc51813ee40 2024-03-29T11:19:27.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.602+0000 7fc50ffff640 10 client.4541.objecter ms_dispatch 0x7fc518000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:27.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.602+0000 7fc50ffff640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:27.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.602+0000 7fc50ffff640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:27.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.603+0000 7fc50ffff640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:19:27.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.603+0000 7fc50ffff640 10 client.4541.objecter ms_handle_connect 0x7fc4f806de30 2024-03-29T11:19:27.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.610+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc51004f010 con 0x7fc51813ee40 2024-03-29T11:19:27.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.610+0000 7fc50ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.610+0000 7fc50ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:27.893 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.916+0000 7fc51d455640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:27.893 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.916+0000 7fc51d455640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:27.893 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.916+0000 7fc51d455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:19:27.893 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.916+0000 7fc51d455640 1 -- 172.21.15.157:0/3820383761 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc518075f10 con 0x7fc51813ee40 2024-03-29T11:19:27.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.916+0000 7fc50ffff640 1 -- 172.21.15.157:0/3820383761 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fc510051110 con 0x7fc51813ee40 2024-03-29T11:19:27.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.916+0000 7fc50ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:27.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.916+0000 7fc50ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:27.894 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:19:27.894 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:27.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.919+0000 7fc51d455640 10 librados: watch_flush enter 2024-03-29T11:19:27.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.919+0000 7fc51d455640 10 librados: watch_flush exit 2024-03-29T11:19:27.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.919+0000 7fc51d455640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:19:27.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.919+0000 7fc51d455640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:19:27.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.920+0000 7fc51d455640 1 -- 172.21.15.157:0/3820383761 >> v1:172.21.15.157:6816/1747058247 conn(0x7fc4f806de30 legacy=0x7fc4f80702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.920+0000 7fc51d455640 10 monclient: shutdown 2024-03-29T11:19:27.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.920+0000 7fc51d455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:27.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.920+0000 7fc51d455640 1 -- 172.21.15.157:0/3820383761 >> v1:172.21.15.157:6790/0 conn(0x7fc51813ee40 legacy=0x7fc51812dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.897 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.920+0000 7fc51d455640 1 -- 172.21.15.157:0/3820383761 shutdown_connections 2024-03-29T11:19:27.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.920+0000 7fc51d455640 1 -- 172.21.15.157:0/3820383761 >> 172.21.15.157:0/3820383761 conn(0x7fc51806b660 msgr2=0x7fc5180a9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:27.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.920+0000 7fc51d455640 1 -- 172.21.15.157:0/3820383761 shutdown_connections 2024-03-29T11:19:27.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.920+0000 7fc51d455640 1 -- 172.21.15.157:0/3820383761 wait complete. 2024-03-29T11:19:27.898 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:27.920+0000 7fc51d455640 1 librados: shutdown 2024-03-29T11:19:27.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:34.720 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:34.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.900+0000 7efc5d5b6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:34.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.900+0000 7efc5d5b6640 10 monclient: build_initial_monmap 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.900+0000 7efc5d5b6640 10 monclient: monmap: 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:19:34.901704+0000 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:19:34.901704+0000 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:19:34.878 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:34.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.901+0000 7efc5d5b6640 1 Processor -- start 2024-03-29T11:19:34.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.901+0000 7efc5d5b6640 1 -- start start 2024-03-29T11:19:34.879 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.901+0000 7efc5d5b6640 10 monclient: init 2024-03-29T11:19:34.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.903+0000 7efc5d5b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:34.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.903+0000 7efc5d5b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:34.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.903+0000 7efc5d5b6640 10 monclient(hunting): picked mon.noname-c con 0x7efc580b1ec0 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:34.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.903+0000 7efc5d5b6640 10 monclient(hunting): picked mon.noname-a con 0x7efc58143c40 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:34.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.903+0000 7efc5d5b6640 10 monclient(hunting): picked mon.noname-b con 0x7efc580b0420 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:34.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.903+0000 7efc5d5b6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580b0940 con 0x7efc58143c40 2024-03-29T11:19:34.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.903+0000 7efc5d5b6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580afb10 con 0x7efc580b1ec0 2024-03-29T11:19:34.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.903+0000 7efc5d5b6640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580afca0 con 0x7efc580b0420 2024-03-29T11:19:34.881 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.903+0000 7efc5d5b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:34.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.903+0000 7efc5d5b6640 10 monclient(hunting): authenticate will time out at 1387.100056s 2024-03-29T11:19:34.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.904+0000 7efc57fff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7efc580b0420 0x7efc58151b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:40736/0 (socket says 172.21.15.157:40736) 2024-03-29T11:19:34.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.904+0000 7efc57fff640 1 -- 172.21.15.157:0/3479417017 learned_addr learned my addr 172.21.15.157:0/3479417017 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:34.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.905+0000 7efc55ffb640 1 -- 172.21.15.157:0/3479417017 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830638341 0 0) 0x7efc580afca0 con 0x7efc580b0420 2024-03-29T11:19:34.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.905+0000 7efc55ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.905+0000 7efc55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.905+0000 7efc55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.905+0000 7efc55ffb640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:19:34.882 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.905+0000 7efc55ffb640 1 -- 172.21.15.157:0/3479417017 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc40003520 con 0x7efc580b0420 2024-03-29T11:19:34.883 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.906+0000 7efc55ffb640 1 -- 172.21.15.157:0/3479417017 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 827493227 0 0) 0x7efc40003520 con 0x7efc580b0420 2024-03-29T11:19:34.883 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.906+0000 7efc55ffb640 1 -- 172.21.15.157:0/3479417017 >> v1:172.21.15.195:6789/0 conn(0x7efc580b1ec0 legacy=0x7efc58143530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:34.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.907+0000 7efc55ffb640 1 -- 172.21.15.157:0/3479417017 >> v1:172.21.15.157:6789/0 conn(0x7efc58143c40 legacy=0x7efc58149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.907+0000 7efc55ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:34.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.907+0000 7efc55ffb640 1 monclient: found mon.noname-b 2024-03-29T11:19:34.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.907+0000 7efc55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:34.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.908+0000 7efc55ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:19:34.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.908+0000 7efc55ffb640 1 -- 172.21.15.157:0/3479417017 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc580b0fb0 con 0x7efc580b0420 2024-03-29T11:19:34.886 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.908+0000 7efc55ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.908+0000 7efc55ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.908+0000 7efc55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.908+0000 7efc55ffb640 1 -- 172.21.15.157:0/3479417017 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7efc50001a10 con 0x7efc580b0420 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.908+0000 7efc55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.908+0000 7efc55ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.908+0000 7efc55ffb640 10 monclient: dump: 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.887 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:34.888 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:34.888 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:34.888 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:34.888 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:34.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.909+0000 7efc5d5b6640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:19:34.888 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.909+0000 7efc5d5b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:34.893 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.909+0000 7efc55ffb640 1 -- 172.21.15.157:0/3479417017 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc50001ca0 con 0x7efc580b0420 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.909+0000 7efc55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.909+0000 7efc55ffb640 1 -- 172.21.15.157:0/3479417017 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7efc50002030 con 0x7efc580b0420 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.909+0000 7efc55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.909+0000 7efc55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.909+0000 7efc55ffb640 10 monclient: dump: 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:34.894 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.909+0000 7efc5d5b6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.909+0000 7efc5d5b6640 4 set_mon_vals no callback set 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.910+0000 7efc5d5b6640 10 monclient: shutdown 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.910+0000 7efc5d5b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.910+0000 7efc5d5b6640 1 -- 172.21.15.157:0/3479417017 >> v1:172.21.15.157:6790/0 conn(0x7efc580b0420 legacy=0x7efc58151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.910+0000 7efc5d5b6640 1 -- 172.21.15.157:0/3479417017 shutdown_connections 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.910+0000 7efc5d5b6640 1 -- 172.21.15.157:0/3479417017 >> 172.21.15.157:0/3479417017 conn(0x7efc5806cb30 msgr2=0x7efc5806cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.911+0000 7efc5d5b6640 1 -- 172.21.15.157:0/3479417017 shutdown_connections 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.911+0000 7efc5d5b6640 1 -- 172.21.15.157:0/3479417017 wait complete. 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.912+0000 7efc5d5b6640 10 monclient: build_initial_monmap 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.912+0000 7efc5d5b6640 10 monclient: monmap: 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:34.895 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.912+0000 7efc5d5b6640 1 librados: starting msgr at 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.912+0000 7efc5d5b6640 1 librados: starting objecter 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.913+0000 7efc5d5b6640 1 Processor -- start 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.913+0000 7efc5d5b6640 1 -- start start 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.913+0000 7efc5d5b6640 1 librados: setting wanted keys 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.913+0000 7efc5d5b6640 1 librados: calling monclient init 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.913+0000 7efc5d5b6640 10 monclient: init 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.914+0000 7efc5d5b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:34.896 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.914+0000 7efc5d5b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:34.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.917+0000 7efc5d5b6640 10 monclient(hunting): picked mon.noname-b con 0x7efc580b0420 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:34.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.917+0000 7efc5d5b6640 10 monclient(hunting): picked mon.noname-a con 0x7efc580b1ec0 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:34.902 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.918+0000 7efc5d5b6640 10 monclient(hunting): picked mon.noname-c con 0x7efc58143c40 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.918+0000 7efc57fff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7efc58143c40 0x7efc58099520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:42308/0 (socket says 172.21.15.157:42308) 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.918+0000 7efc57fff640 1 -- 172.21.15.157:0/1915403266 learned_addr learned my addr 172.21.15.157:0/1915403266 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.918+0000 7efc56ffd640 1 --1- 172.21.15.157:0/1915403266 >> v1:172.21.15.195:6789/0 conn(0x7efc580b0420 0x7efc581485d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:58287/0 (socket says 172.21.15.157:58287) 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.919+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc58149710 con 0x7efc580b1ec0 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.919+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc581e3f30 con 0x7efc580b0420 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.919+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc58148ce0 con 0x7efc58143c40 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.919+0000 7efc5d5b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.919+0000 7efc5d5b6640 10 monclient(hunting): authenticate will time out at 1387.116259s 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.919+0000 7efc3f7fe640 10 client.?.objecter ms_handle_connect 0x7efc58143c40 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.919+0000 7efc3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.919+0000 7efc3f7fe640 10 client.?.objecter ms_handle_connect 0x7efc580b0420 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.919+0000 7efc3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.920+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89772404 0 0) 0x7efc581e3f30 con 0x7efc580b0420 2024-03-29T11:19:34.903 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.920+0000 7efc3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.920+0000 7efc3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.920+0000 7efc3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.920+0000 7efc3f7fe640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.920+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc44004eb0 con 0x7efc580b0420 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101487674 0 0) 0x7efc58148ce0 con 0x7efc58143c40 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc581e3f30 con 0x7efc58143c40 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 client.?.objecter ms_handle_connect 0x7efc580b1ec0 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.904 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2250028484 0 0) 0x7efc581e3f30 con 0x7efc58143c40 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc58148ce0 con 0x7efc58143c40 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7efc500018d0 con 0x7efc58143c40 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.905 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603558269 0 0) 0x7efc58149710 con 0x7efc580b1ec0 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc581e3f30 con 0x7efc580b1ec0 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 501926140 0 0) 0x7efc58148ce0 con 0x7efc58143c40 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 >> v1:172.21.15.195:6789/0 conn(0x7efc580b0420 legacy=0x7efc581485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.906 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 >> v1:172.21.15.157:6789/0 conn(0x7efc580b1ec0 legacy=0x7efc581e4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 monclient: found mon.c 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5806fed0 con 0x7efc58143c40 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc5d5b6640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc5d5b6640 10 monclient: _renew_subs 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc5d5b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:19:34.907 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc581e4b60 con 0x7efc58143c40 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc5d5b6640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc5d5b6640 10 monclient: _renew_subs 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc5d5b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc58099e10 con 0x7efc58143c40 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.921+0000 7efc5d5b6640 1 librados: init done 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc50001a30 con 0x7efc58143c40 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7efc50005110 con 0x7efc58143c40 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc3f7fe640 10 monclient: dump: 2024-03-29T11:19:34.908 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc5d5b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc5d5b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc5d5b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc58148e70 con 0x7efc58143c40 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.923+0000 7efc557fa640 4 set_mon_vals no callback set 2024-03-29T11:19:34.909 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.924+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7efc5001b0e0 con 0x7efc58143c40 2024-03-29T11:19:34.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.925+0000 7efc56ffd640 10 monclient: get_auth_request con 0x7efc4406df00 auth_method 0 2024-03-29T11:19:34.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.925+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7efc50084d90 con 0x7efc58143c40 2024-03-29T11:19:34.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.925+0000 7efc3f7fe640 10 client.4547.objecter ms_dispatch 0x7efc58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:34.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.925+0000 7efc3f7fe640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:34.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.925+0000 7efc3f7fe640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:34.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.925+0000 7efc3f7fe640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:19:34.910 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.925+0000 7efc3f7fe640 10 client.4547.objecter ms_handle_connect 0x7efc4406df00 2024-03-29T11:19:34.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.932+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc58148e70 con 0x7efc58143c40 2024-03-29T11:19:34.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.932+0000 7efc3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.911 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:34.932+0000 7efc3f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:35.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.257+0000 7efc5d5b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:35.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.257+0000 7efc5d5b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:35.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.257+0000 7efc5d5b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:19:35.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.257+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc58148e70 con 0x7efc58143c40 2024-03-29T11:19:35.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.260+0000 7efc3f7fe640 1 -- 172.21.15.157:0/1915403266 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7efc58148e70 con 0x7efc58143c40 2024-03-29T11:19:35.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.260+0000 7efc3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:35.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.260+0000 7efc3f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:35.238 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:19:35.238 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:35.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.264+0000 7efc5d5b6640 10 librados: watch_flush enter 2024-03-29T11:19:35.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.264+0000 7efc5d5b6640 10 librados: watch_flush exit 2024-03-29T11:19:35.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.264+0000 7efc5d5b6640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:19:35.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.264+0000 7efc5d5b6640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:19:35.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.265+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 >> v1:172.21.15.157:6816/1747058247 conn(0x7efc4406df00 legacy=0x7efc440703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.242 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.265+0000 7efc5d5b6640 10 monclient: shutdown 2024-03-29T11:19:35.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.265+0000 7efc5d5b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:35.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.265+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 >> v1:172.21.15.157:6790/0 conn(0x7efc58143c40 legacy=0x7efc58099520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.265+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 shutdown_connections 2024-03-29T11:19:35.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.265+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 >> 172.21.15.157:0/1915403266 conn(0x7efc5806cb30 msgr2=0x7efc5813ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:35.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.266+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 shutdown_connections 2024-03-29T11:19:35.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.266+0000 7efc5d5b6640 1 -- 172.21.15.157:0/1915403266 wait complete. 2024-03-29T11:19:35.243 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:35.266+0000 7efc5d5b6640 1 librados: shutdown 2024-03-29T11:19:35.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:42.067 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:42.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.242+0000 7f4201cba640 10 monclient: get_monmap_and_config 2024-03-29T11:19:42.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.242+0000 7f4201cba640 10 monclient: build_initial_monmap 2024-03-29T11:19:42.219 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.242+0000 7f4201cba640 10 monclient: monmap: 2024-03-29T11:19:42.219 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:42.219 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:42.219 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:19:42.243513+0000 2024-03-29T11:19:42.219 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:19:42.243513+0000 2024-03-29T11:19:42.220 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:42.220 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:42.220 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:42.220 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:19:42.220 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:19:42.220 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:42.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.247+0000 7f4201cba640 1 Processor -- start 2024-03-29T11:19:42.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.247+0000 7f4201cba640 1 -- start start 2024-03-29T11:19:42.224 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.247+0000 7f4201cba640 10 monclient: init 2024-03-29T11:19:42.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.248+0000 7f4201cba640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:42.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.248+0000 7f4201cba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:42.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.248+0000 7f4201cba640 10 monclient(hunting): picked mon.noname-a con 0x7f41fc1457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:42.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.250+0000 7f4201cba640 10 monclient(hunting): picked mon.noname-b con 0x7f41fc13f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:42.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.250+0000 7f4201cba640 10 monclient(hunting): picked mon.noname-c con 0x7f41fc0a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:42.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.250+0000 7f4201cba640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc0aaef0 con 0x7f41fc1457f0 2024-03-29T11:19:42.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.250+0000 7f4201cba640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc145bd0 con 0x7f41fc0a9d70 2024-03-29T11:19:42.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.250+0000 7f4201cba640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc145eb0 con 0x7f41fc13f650 2024-03-29T11:19:42.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.250+0000 7f4201cba640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:42.227 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.250+0000 7f4201cba640 10 monclient(hunting): authenticate will time out at 1394.446951s 2024-03-29T11:19:42.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.254+0000 7f41fb7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f41fc1457f0 0x7f41fc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:38819/0 (socket says 172.21.15.157:38819) 2024-03-29T11:19:42.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.254+0000 7f41fb7fe640 1 -- 172.21.15.157:0/969780181 learned_addr learned my addr 172.21.15.157:0/969780181 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:42.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.254+0000 7f41fa7fc640 1 -- 172.21.15.157:0/969780181 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703496768 0 0) 0x7f41fc0aaef0 con 0x7f41fc1457f0 2024-03-29T11:19:42.231 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.254+0000 7f41fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:19:42.232 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 1 -- 172.21.15.157:0/969780181 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e8003520 con 0x7f41fc1457f0 2024-03-29T11:19:42.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 1 -- 172.21.15.157:0/969780181 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136286936 0 0) 0x7f41e8003520 con 0x7f41fc1457f0 2024-03-29T11:19:42.233 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 1 -- 172.21.15.157:0/969780181 >> v1:172.21.15.157:6790/0 conn(0x7f41fc13f650 legacy=0x7f41fc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 1 -- 172.21.15.157:0/969780181 >> v1:172.21.15.195:6789/0 conn(0x7f41fc0a9d70 legacy=0x7f41fc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 1 monclient: found mon.noname-a 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 1 -- 172.21.15.157:0/969780181 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc146340 con 0x7f41fc1457f0 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 1 -- 172.21.15.157:0/969780181 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f41f4001a10 con 0x7f41fc1457f0 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f4201cba640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f4201cba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.234 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:42.235 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient: dump: 2024-03-29T11:19:42.235 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:42.235 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:42.235 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:42.235 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:42.235 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.235 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:42.235 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:42.235 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:42.235 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f4201cba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 1 -- 172.21.15.157:0/969780181 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41f4001be0 con 0x7f41fc1457f0 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f41fa7fc640 1 -- 172.21.15.157:0/969780181 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f41f4002030 con 0x7f41fc1457f0 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f4201cba640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.255+0000 7f4201cba640 4 set_mon_vals no callback set 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.256+0000 7f41fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.256+0000 7f41fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.256+0000 7f41fa7fc640 10 monclient: dump: 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:42.236 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.237 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:42.237 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:42.237 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:42.237 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:42.237 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:42.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.256+0000 7f4201cba640 10 monclient: shutdown 2024-03-29T11:19:42.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.257+0000 7f4201cba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.257+0000 7f4201cba640 1 -- 172.21.15.157:0/969780181 >> v1:172.21.15.157:6789/0 conn(0x7f41fc1457f0 legacy=0x7f41fc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.257+0000 7f4201cba640 1 -- 172.21.15.157:0/969780181 shutdown_connections 2024-03-29T11:19:42.237 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.257+0000 7f4201cba640 1 -- 172.21.15.157:0/969780181 >> 172.21.15.157:0/969780181 conn(0x7f41fc066a00 msgr2=0x7f41fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.257+0000 7f4201cba640 1 -- 172.21.15.157:0/969780181 shutdown_connections 2024-03-29T11:19:42.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.257+0000 7f4201cba640 1 -- 172.21.15.157:0/969780181 wait complete. 2024-03-29T11:19:42.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.258+0000 7f4201cba640 10 monclient: build_initial_monmap 2024-03-29T11:19:42.238 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.258+0000 7f4201cba640 10 monclient: monmap: 2024-03-29T11:19:42.238 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:42.238 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:42.238 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:19:42.238 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:19:42.239 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:42.239 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:42.239 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:42.239 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:19:42.239 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:19:42.239 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:42.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.259+0000 7f4201cba640 1 librados: starting msgr at 2024-03-29T11:19:42.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.259+0000 7f4201cba640 1 librados: starting objecter 2024-03-29T11:19:42.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.260+0000 7f4201cba640 1 Processor -- start 2024-03-29T11:19:42.239 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.260+0000 7f4201cba640 1 -- start start 2024-03-29T11:19:42.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.260+0000 7f4201cba640 1 librados: setting wanted keys 2024-03-29T11:19:42.240 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.260+0000 7f4201cba640 1 librados: calling monclient init 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.260+0000 7f4201cba640 10 monclient: init 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.261+0000 7f4201cba640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.261+0000 7f4201cba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.261+0000 7f4201cba640 10 monclient(hunting): picked mon.noname-b con 0x7f41fc0a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.261+0000 7f4201cba640 10 monclient(hunting): picked mon.noname-a con 0x7f41fc13f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.261+0000 7f4201cba640 10 monclient(hunting): picked mon.noname-c con 0x7f41fc1457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.261+0000 7f4201cba640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc140940 con 0x7f41fc13f650 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.261+0000 7f4201cba640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc140c20 con 0x7f41fc0a9d70 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.261+0000 7f4201cba640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc140720 con 0x7f41fc1457f0 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.261+0000 7f4201cba640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:42.241 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.261+0000 7f4201cba640 10 monclient(hunting): authenticate will time out at 1394.458531s 2024-03-29T11:19:42.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.269+0000 7f41faffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f41fc13f650 0x7f41fc1d7680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:39041/0 (socket says 172.21.15.157:39041) 2024-03-29T11:19:42.246 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.269+0000 7f41faffd640 1 -- 172.21.15.157:0/644231047 learned_addr learned my addr 172.21.15.157:0/644231047 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:42.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.270+0000 7f41f3fff640 10 client.?.objecter ms_handle_connect 0x7f41fc0a9d70 2024-03-29T11:19:42.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.270+0000 7f41f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.270+0000 7f41f3fff640 10 client.?.objecter ms_handle_connect 0x7f41fc13f650 2024-03-29T11:19:42.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.270+0000 7f41f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.270+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422977189 0 0) 0x7f41fc140940 con 0x7f41fc13f650 2024-03-29T11:19:42.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.248 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:19:42.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41dc004eb0 con 0x7f41fc13f650 2024-03-29T11:19:42.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239160143 0 0) 0x7f41fc140c20 con 0x7f41fc0a9d70 2024-03-29T11:19:42.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:19:42.249 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.271+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41fc140940 con 0x7f41fc0a9d70 2024-03-29T11:19:42.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 10 client.?.objecter ms_handle_connect 0x7f41fc1457f0 2024-03-29T11:19:42.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3044718922 0 0) 0x7f41dc004eb0 con 0x7f41fc13f650 2024-03-29T11:19:42.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41fc140c20 con 0x7f41fc13f650 2024-03-29T11:19:42.252 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f41ec001960 con 0x7f41fc13f650 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 10 monclient(hunting): dump: 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2421096768 0 0) 0x7f41fc140940 con 0x7f41fc0a9d70 2024-03-29T11:19:42.253 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41dc004eb0 con 0x7f41fc0a9d70 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f41f4001ef0 con 0x7f41fc0a9d70 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.272+0000 7f41f3fff640 10 monclient(hunting): dump: 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:42.254 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 595001772 0 0) 0x7f41fc140c20 con 0x7f41fc13f650 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 >> v1:172.21.15.157:6790/0 conn(0x7f41fc1457f0 legacy=0x7f41fc0b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 >> v1:172.21.15.195:6789/0 conn(0x7f41fc0a9d70 legacy=0x7f41fc140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 1 monclient: found mon.a 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc1d7d90 con 0x7f41fc13f650 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.273+0000 7f41f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.274+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41ec0035f0 con 0x7f41fc13f650 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.274+0000 7f41f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:42.255 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.274+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f41ec003940 con 0x7f41fc13f650 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.274+0000 7f41f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.274+0000 7f41f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.274+0000 7f41f3fff640 10 monclient: dump: 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:42.256 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.275+0000 7f41f9ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.276+0000 7f4201cba640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.276+0000 7f4201cba640 10 monclient: _renew_subs 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.276+0000 7f4201cba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.276+0000 7f4201cba640 1 -- 172.21.15.157:0/644231047 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41fc1410b0 con 0x7f41fc13f650 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.276+0000 7f4201cba640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.276+0000 7f4201cba640 10 monclient: _renew_subs 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.276+0000 7f4201cba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.276+0000 7f4201cba640 1 -- 172.21.15.157:0/644231047 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41fc0b0d80 con 0x7f41fc13f650 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.276+0000 7f4201cba640 1 librados: init done 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.278+0000 7f4201cba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.278+0000 7f4201cba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.278+0000 7f4201cba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:19:42.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.278+0000 7f4201cba640 1 -- 172.21.15.157:0/644231047 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41fc075f10 con 0x7f41fc13f650 2024-03-29T11:19:42.260 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.284+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f41ec003aa0 con 0x7f41fc13f650 2024-03-29T11:19:42.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.285+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f41ec0849e0 con 0x7f41fc13f650 2024-03-29T11:19:42.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.285+0000 7f41f3fff640 10 client.4512.objecter ms_dispatch 0x7f41fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:42.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.285+0000 7f41f3fff640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:42.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.285+0000 7f41f3fff640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:42.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.285+0000 7f41f3fff640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T11:19:42.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.285+0000 7f41fb7fe640 10 monclient: get_auth_request con 0x7f41dc06de90 auth_method 0 2024-03-29T11:19:42.263 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.286+0000 7f41f3fff640 10 client.4512.objecter ms_handle_connect 0x7f41dc06de90 2024-03-29T11:19:42.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.291+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41ec0502c0 con 0x7f41fc13f650 2024-03-29T11:19:42.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.291+0000 7f41f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.291+0000 7f41f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:42.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.612+0000 7f4201cba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.612+0000 7f4201cba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.612+0000 7f4201cba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:19:42.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.612+0000 7f4201cba640 1 -- 172.21.15.157:0/644231047 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41fc04ef70 con 0x7f41fc13f650 2024-03-29T11:19:42.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.613+0000 7f41f3fff640 1 -- 172.21.15.157:0/644231047 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f41ec0521f0 con 0x7f41fc13f650 2024-03-29T11:19:42.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.613+0000 7f41f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.613+0000 7f41f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:42.591 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:19:42.591 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:42.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.617+0000 7f4201cba640 10 librados: watch_flush enter 2024-03-29T11:19:42.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.617+0000 7f4201cba640 10 librados: watch_flush exit 2024-03-29T11:19:42.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.617+0000 7f4201cba640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T11:19:42.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.617+0000 7f4201cba640 10 client.4512.objecter successfully canceled tick 2024-03-29T11:19:42.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.617+0000 7f4201cba640 1 -- 172.21.15.157:0/644231047 >> v1:172.21.15.157:6816/1747058247 conn(0x7f41dc06de90 legacy=0x7f41dc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.617+0000 7f4201cba640 10 monclient: shutdown 2024-03-29T11:19:42.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.617+0000 7f4201cba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.617+0000 7f4201cba640 1 -- 172.21.15.157:0/644231047 >> v1:172.21.15.157:6789/0 conn(0x7f41fc13f650 legacy=0x7f41fc1d7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.618+0000 7f4201cba640 1 -- 172.21.15.157:0/644231047 shutdown_connections 2024-03-29T11:19:42.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.618+0000 7f4201cba640 1 -- 172.21.15.157:0/644231047 >> 172.21.15.157:0/644231047 conn(0x7f41fc066a00 msgr2=0x7f41fc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.618+0000 7f4201cba640 1 -- 172.21.15.157:0/644231047 shutdown_connections 2024-03-29T11:19:42.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.618+0000 7f4201cba640 1 -- 172.21.15.157:0/644231047 wait complete. 2024-03-29T11:19:42.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:42.618+0000 7f4201cba640 1 librados: shutdown 2024-03-29T11:19:42.612 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:49.420 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:49.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.595+0000 7f6ba5ea8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:49.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.595+0000 7f6ba5ea8640 10 monclient: build_initial_monmap 2024-03-29T11:19:49.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.595+0000 7f6ba5ea8640 10 monclient: monmap: 2024-03-29T11:19:49.573 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:49.573 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:49.573 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:19:49.596241+0000 2024-03-29T11:19:49.573 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:19:49.596241+0000 2024-03-29T11:19:49.573 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:49.573 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:49.573 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:49.573 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:19:49.573 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:19:49.573 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:49.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.597+0000 7f6ba5ea8640 1 Processor -- start 2024-03-29T11:19:49.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.597+0000 7f6ba5ea8640 1 -- start start 2024-03-29T11:19:49.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.597+0000 7f6ba5ea8640 10 monclient: init 2024-03-29T11:19:49.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.598+0000 7f6ba5ea8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:49.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.598+0000 7f6ba5ea8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:49.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.598+0000 7f6ba5ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ba00b0860 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:49.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.598+0000 7f6ba5ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba013ef30 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:49.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.601+0000 7f6ba5ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ba00af8e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:49.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.601+0000 7f6b9ffff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f6ba00af8e0 0x7f6ba0149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:51459/0 (socket says 172.21.15.157:51459) 2024-03-29T11:19:49.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.601+0000 7f6b9ffff640 1 -- 172.21.15.157:0/196267109 learned_addr learned my addr 172.21.15.157:0/196267109 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:49.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.601+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/196267109 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba00b0c40 con 0x7f6ba00b0860 2024-03-29T11:19:49.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.601+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/196267109 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba00aed10 con 0x7f6ba013ef30 2024-03-29T11:19:49.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.601+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/196267109 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba00afd50 con 0x7f6ba00af8e0 2024-03-29T11:19:49.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.601+0000 7f6ba5ea8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:49.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.601+0000 7f6ba5ea8640 10 monclient(hunting): authenticate will time out at 1401.798760s 2024-03-29T11:19:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.602+0000 7f6b9e7fc640 1 -- 172.21.15.157:0/196267109 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606356417 0 0) 0x7f6ba00afd50 con 0x7f6ba00af8e0 2024-03-29T11:19:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.602+0000 7f6b9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.602+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.602+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.602+0000 7f6b9e7fc640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:19:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.602+0000 7f6b9e7fc640 1 -- 172.21.15.157:0/196267109 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b88003520 con 0x7f6ba00af8e0 2024-03-29T11:19:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.602+0000 7f6b9e7fc640 1 -- 172.21.15.157:0/196267109 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361845552 0 0) 0x7f6ba00b0c40 con 0x7f6ba00b0860 2024-03-29T11:19:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.581 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 1 -- 172.21.15.157:0/196267109 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ba00afd50 con 0x7f6ba00b0860 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 1 -- 172.21.15.157:0/196267109 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1401039266 0 0) 0x7f6b88003520 con 0x7f6ba00af8e0 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 1 -- 172.21.15.157:0/196267109 >> v1:172.21.15.195:6789/0 conn(0x7f6ba013ef30 legacy=0x7f6ba00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 1 -- 172.21.15.157:0/196267109 >> v1:172.21.15.157:6789/0 conn(0x7f6ba00b0860 legacy=0x7f6ba013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 1 -- 172.21.15.157:0/196267109 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba00ab1b0 con 0x7f6ba00af8e0 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:49.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 1 -- 172.21.15.157:0/196267109 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6b98001990 con 0x7f6ba00af8e0 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6ba5ea8640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6ba5ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.603+0000 7f6b9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.604+0000 7f6b9e7fc640 10 monclient: dump: 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:49.583 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:49.584 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:49.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.604+0000 7f6ba5ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:49.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.604+0000 7f6b9e7fc640 1 -- 172.21.15.157:0/196267109 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b98001bb0 con 0x7f6ba00af8e0 2024-03-29T11:19:49.584 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.604+0000 7f6b9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:49.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.604+0000 7f6ba5ea8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:49.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.604+0000 7f6ba5ea8640 4 set_mon_vals no callback set 2024-03-29T11:19:49.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.605+0000 7f6ba5ea8640 10 monclient: shutdown 2024-03-29T11:19:49.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.605+0000 7f6ba5ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:49.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.605+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/196267109 >> v1:172.21.15.157:6790/0 conn(0x7f6ba00af8e0 legacy=0x7f6ba0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.605+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/196267109 shutdown_connections 2024-03-29T11:19:49.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.605+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/196267109 >> 172.21.15.157:0/196267109 conn(0x7f6ba006b660 msgr2=0x7f6ba006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:49.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.606+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/196267109 shutdown_connections 2024-03-29T11:19:49.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.609+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/196267109 wait complete. 2024-03-29T11:19:49.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.610+0000 7f6ba5ea8640 10 monclient: build_initial_monmap 2024-03-29T11:19:49.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.610+0000 7f6ba5ea8640 10 monclient: monmap: 2024-03-29T11:19:49.588 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:49.588 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:49.588 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:19:49.588 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:19:49.588 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:49.588 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:49.588 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:49.588 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:19:49.589 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:19:49.589 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:49.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.611+0000 7f6ba5ea8640 1 librados: starting msgr at 2024-03-29T11:19:49.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.611+0000 7f6ba5ea8640 1 librados: starting objecter 2024-03-29T11:19:49.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.611+0000 7f6ba5ea8640 1 Processor -- start 2024-03-29T11:19:49.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.614+0000 7f6ba5ea8640 1 -- start start 2024-03-29T11:19:49.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.614+0000 7f6ba5ea8640 1 librados: setting wanted keys 2024-03-29T11:19:49.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.614+0000 7f6ba5ea8640 1 librados: calling monclient init 2024-03-29T11:19:49.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.614+0000 7f6ba5ea8640 10 monclient: init 2024-03-29T11:19:49.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.615+0000 7f6ba5ea8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:49.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.615+0000 7f6ba5ea8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:49.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.615+0000 7f6ba5ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ba00af8e0 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:49.593 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.615+0000 7f6ba5ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba00b0860 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:49.593 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.615+0000 7f6ba5ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ba013ef30 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:49.593 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.615+0000 7f6ba5ea8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba01361a0 con 0x7f6ba00af8e0 2024-03-29T11:19:49.593 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.615+0000 7f6ba5ea8640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba0139ce0 con 0x7f6ba013ef30 2024-03-29T11:19:49.593 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.615+0000 7f6ba5ea8640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba01399f0 con 0x7f6ba00b0860 2024-03-29T11:19:49.593 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.615+0000 7f6ba5ea8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:49.593 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.615+0000 7f6ba5ea8640 10 monclient(hunting): authenticate will time out at 1401.812081s 2024-03-29T11:19:49.595 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.618+0000 7f6b9f7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6ba00af8e0 0x7f6ba01392e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:52039/0 (socket says 172.21.15.157:52039) 2024-03-29T11:19:49.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.618+0000 7f6b9f7fe640 1 -- 172.21.15.157:0/2811836747 learned_addr learned my addr 172.21.15.157:0/2811836747 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:49.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.619+0000 7f6b87fff640 10 client.?.objecter ms_handle_connect 0x7f6ba00af8e0 2024-03-29T11:19:49.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.619+0000 7f6b87fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:49.596 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.619+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516339825 0 0) 0x7f6ba01361a0 con 0x7f6ba00af8e0 2024-03-29T11:19:49.597 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.619+0000 7f6b87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.597 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.619+0000 7f6b87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.597 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.619+0000 7f6b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.597 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.619+0000 7f6b87fff640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:19:49.597 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.619+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b8c004eb0 con 0x7f6ba00af8e0 2024-03-29T11:19:49.601 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.621+0000 7f6b87fff640 10 client.?.objecter ms_handle_connect 0x7f6ba013ef30 2024-03-29T11:19:49.601 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.621+0000 7f6b87fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:49.601 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.621+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399461272 0 0) 0x7f6ba0139ce0 con 0x7f6ba013ef30 2024-03-29T11:19:49.601 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.621+0000 7f6b87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.601 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.621+0000 7f6b87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.601 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.621+0000 7f6b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.621+0000 7f6b87fff640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.621+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ba01361a0 con 0x7f6ba013ef30 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.622+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2443373566 0 0) 0x7f6ba01361a0 con 0x7f6ba013ef30 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.622+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ba0139ce0 con 0x7f6ba013ef30 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.622+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6b98001460 con 0x7f6ba013ef30 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.622+0000 7f6b87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.622+0000 7f6b87fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.622+0000 7f6b87fff640 10 monclient(hunting): dump: 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.602 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.622+0000 7f6b87fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.622+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2909895269 0 0) 0x7f6ba0139ce0 con 0x7f6ba013ef30 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.622+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 >> v1:172.21.15.157:6790/0 conn(0x7f6ba00b0860 legacy=0x7f6ba013ab40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 >> v1:172.21.15.157:6789/0 conn(0x7f6ba00af8e0 legacy=0x7f6ba01392e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6b87fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6b87fff640 1 monclient: found mon.b 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6b87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6b87fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba0131e70 con 0x7f6ba013ef30 2024-03-29T11:19:49.603 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6b87fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6b87fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6b87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6ba5ea8640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6ba5ea8640 10 monclient: _renew_subs 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6ba5ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/2811836747 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba01320c0 con 0x7f6ba013ef30 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6ba5ea8640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6ba5ea8640 10 monclient: _renew_subs 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6ba5ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/2811836747 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba01e31f0 con 0x7f6ba013ef30 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.623+0000 7f6ba5ea8640 1 librados: init done 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.624+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b980050e0 con 0x7f6ba013ef30 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.624+0000 7f6b87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:49.604 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.624+0000 7f6b9dffb640 4 set_mon_vals no callback set 2024-03-29T11:19:49.605 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.625+0000 7f6ba5ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:49.605 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.626+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6b98005470 con 0x7f6ba013ef30 2024-03-29T11:19:49.605 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.626+0000 7f6b87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.605 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.626+0000 7f6b87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:49.605 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.626+0000 7f6b87fff640 10 monclient: dump: 2024-03-29T11:19:49.605 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:49.605 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.627+0000 7f6ba5ea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.627+0000 7f6ba5ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:49.606 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.627+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/2811836747 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b80005080 con 0x7f6ba013ef30 2024-03-29T11:19:49.607 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.630+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f6b98001da0 con 0x7f6ba013ef30 2024-03-29T11:19:49.608 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.631+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f6b98084e20 con 0x7f6ba013ef30 2024-03-29T11:19:49.608 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.631+0000 7f6b87fff640 10 client.4513.objecter ms_dispatch 0x7f6ba0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:49.608 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.631+0000 7f6b87fff640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:49.608 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.631+0000 7f6b87fff640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:49.608 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.631+0000 7f6b87fff640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-29T11:19:49.608 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.631+0000 7f6b9f7fe640 10 monclient: get_auth_request con 0x7f6b8c06dce0 auth_method 0 2024-03-29T11:19:49.608 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.631+0000 7f6b87fff640 10 client.4513.objecter ms_handle_connect 0x7f6b8c06dce0 2024-03-29T11:19:49.615 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.638+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b980507c0 con 0x7f6ba013ef30 2024-03-29T11:19:49.615 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.638+0000 7f6b87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:49.615 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.638+0000 7f6b87fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:49.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.962+0000 7f6ba5ea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:49.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.962+0000 7f6ba5ea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:49.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.962+0000 7f6ba5ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:49.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.962+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/2811836747 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b80005330 con 0x7f6ba013ef30 2024-03-29T11:19:49.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.965+0000 7f6b87fff640 1 -- 172.21.15.157:0/2811836747 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f6b980526f0 con 0x7f6ba013ef30 2024-03-29T11:19:49.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.965+0000 7f6b87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:49.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.965+0000 7f6b87fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:49.947 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:19:49.947 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:49.949 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 10 librados: watch_flush enter 2024-03-29T11:19:49.949 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 10 librados: watch_flush exit 2024-03-29T11:19:49.949 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-29T11:19:49.949 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 10 client.4513.objecter successfully canceled tick 2024-03-29T11:19:49.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/2811836747 >> v1:172.21.15.157:6816/1747058247 conn(0x7f6b8c06dce0 legacy=0x7f6b8c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 10 monclient: shutdown 2024-03-29T11:19:49.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:49.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/2811836747 >> v1:172.21.15.195:6789/0 conn(0x7f6ba013ef30 legacy=0x7f6ba013b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/2811836747 shutdown_connections 2024-03-29T11:19:49.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/2811836747 >> 172.21.15.157:0/2811836747 conn(0x7f6ba006b660 msgr2=0x7f6ba00ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:49.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.972+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/2811836747 shutdown_connections 2024-03-29T11:19:49.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.973+0000 7f6ba5ea8640 1 -- 172.21.15.157:0/2811836747 wait complete. 2024-03-29T11:19:49.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:49.973+0000 7f6ba5ea8640 1 librados: shutdown 2024-03-29T11:19:49.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:56.778 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:56.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.954+0000 7f7cdfb4b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:56.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.954+0000 7f7cdfb4b640 10 monclient: build_initial_monmap 2024-03-29T11:19:56.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.954+0000 7f7cdfb4b640 10 monclient: monmap: 2024-03-29T11:19:56.931 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:56.931 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:56.931 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:19:56.955357+0000 2024-03-29T11:19:56.932 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:19:56.955357+0000 2024-03-29T11:19:56.932 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:56.932 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:56.932 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:56.932 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:19:56.932 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:19:56.932 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:56.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.955+0000 7f7cdfb4b640 1 Processor -- start 2024-03-29T11:19:56.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.955+0000 7f7cdfb4b640 1 -- start start 2024-03-29T11:19:56.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.955+0000 7f7cdfb4b640 10 monclient: init 2024-03-29T11:19:56.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.956+0000 7f7cdfb4b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:56.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.956+0000 7f7cdfb4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:56.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.960+0000 7f7cdfb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f7cd81457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:56.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.960+0000 7f7cdfb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f7cd813f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:56.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.960+0000 7f7cdfb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f7cd80a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:56.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.960+0000 7f7cdfb4b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80aaef0 con 0x7f7cd81457f0 2024-03-29T11:19:56.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.960+0000 7f7cdfb4b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd8145bd0 con 0x7f7cd80a9d70 2024-03-29T11:19:56.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.960+0000 7f7cdfb4b640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd8145eb0 con 0x7f7cd813f650 2024-03-29T11:19:56.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.960+0000 7f7cdfb4b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.960+0000 7f7cdfb4b640 10 monclient(hunting): authenticate will time out at 1409.157366s 2024-03-29T11:19:56.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.960+0000 7f7cdd8e7640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f7cd81457f0 0x7f7cd813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:56547/0 (socket says 172.21.15.157:56547) 2024-03-29T11:19:56.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.960+0000 7f7cdd8e7640 1 -- 172.21.15.157:0/2981184042 learned_addr learned my addr 172.21.15.157:0/2981184042 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:56.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.961+0000 7f7cdc8e5640 1 -- 172.21.15.157:0/2981184042 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667977747 0 0) 0x7f7cd80aaef0 con 0x7f7cd81457f0 2024-03-29T11:19:56.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.961+0000 7f7cdc8e5640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.961+0000 7f7cdc8e5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.961+0000 7f7cdc8e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.961+0000 7f7cdc8e5640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:19:56.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.961+0000 7f7cdc8e5640 1 -- 172.21.15.157:0/2981184042 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc4003520 con 0x7f7cd81457f0 2024-03-29T11:19:56.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.963+0000 7f7cdc8e5640 1 -- 172.21.15.157:0/2981184042 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1000214820 0 0) 0x7f7cc4003520 con 0x7f7cd81457f0 2024-03-29T11:19:56.939 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.963+0000 7f7cdc8e5640 1 -- 172.21.15.157:0/2981184042 >> v1:172.21.15.157:6790/0 conn(0x7f7cd813f650 legacy=0x7f7cd80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:56.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.964+0000 7f7cdc8e5640 1 -- 172.21.15.157:0/2981184042 >> v1:172.21.15.195:6789/0 conn(0x7f7cd80a9d70 legacy=0x7f7cd814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:56.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 10 monclient: _finish_hunting 0 2024-03-29T11:19:56.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 1 monclient: found mon.noname-a 2024-03-29T11:19:56.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:56.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:19:56.946 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 1 -- 172.21.15.157:0/2981184042 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd8146340 con 0x7f7cd81457f0 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 10 monclient: _finish_auth 0 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 10 monclient: _check_auth_tickets 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 1 -- 172.21.15.157:0/2981184042 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7cd4001690 con 0x7f7cd81457f0 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 10 monclient: dump: 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:56.947 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdfb4b640 5 monclient: authenticate success, global_id 4521 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdfb4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 1 -- 172.21.15.157:0/2981184042 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cd4001860 con 0x7f7cd81457f0 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdc8e5640 1 -- 172.21.15.157:0/2981184042 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7cd4001cb0 con 0x7f7cd81457f0 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.966+0000 7f7cdfb4b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.967+0000 7f7cdfb4b640 4 set_mon_vals no callback set 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.968+0000 7f7cdfb4b640 10 monclient: shutdown 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.968+0000 7f7cdfb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.968+0000 7f7cdfb4b640 1 -- 172.21.15.157:0/2981184042 >> v1:172.21.15.157:6789/0 conn(0x7f7cd81457f0 legacy=0x7f7cd813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.968+0000 7f7cdc8e5640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:19:56.949 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.968+0000 7f7cdfb4b640 1 -- 172.21.15.157:0/2981184042 shutdown_connections 2024-03-29T11:19:56.949 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.968+0000 7f7cdfb4b640 1 -- 172.21.15.157:0/2981184042 >> 172.21.15.157:0/2981184042 conn(0x7f7cd8066a00 msgr2=0x7f7cd8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:56.949 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.969+0000 7f7cdfb4b640 1 -- 172.21.15.157:0/2981184042 shutdown_connections 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.970+0000 7f7cdfb4b640 1 -- 172.21.15.157:0/2981184042 wait complete. 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.970+0000 7f7cdfb4b640 10 monclient: build_initial_monmap 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.970+0000 7f7cdfb4b640 10 monclient: monmap: 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:19:56.950 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:19:56.951 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:56.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.971+0000 7f7cdfb4b640 1 librados: starting msgr at 2024-03-29T11:19:56.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.971+0000 7f7cdfb4b640 1 librados: starting objecter 2024-03-29T11:19:56.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.971+0000 7f7cdfb4b640 1 Processor -- start 2024-03-29T11:19:56.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.971+0000 7f7cdfb4b640 1 -- start start 2024-03-29T11:19:56.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.971+0000 7f7cdfb4b640 1 librados: setting wanted keys 2024-03-29T11:19:56.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.971+0000 7f7cdfb4b640 1 librados: calling monclient init 2024-03-29T11:19:56.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.971+0000 7f7cdfb4b640 10 monclient: init 2024-03-29T11:19:56.951 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.972+0000 7f7cdfb4b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:56.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.973+0000 7f7cdfb4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:56.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.973+0000 7f7cdfb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f7cd80a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:19:56.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.973+0000 7f7cdfb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f7cd813f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:19:56.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.973+0000 7f7cdfb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f7cd8141860 addr v1:172.21.15.195:6789/0 2024-03-29T11:19:56.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.973+0000 7f7cdfb4b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd8137790 con 0x7f7cd813f650 2024-03-29T11:19:56.953 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.973+0000 7f7cdfb4b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd806e7d0 con 0x7f7cd8141860 2024-03-29T11:19:56.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.973+0000 7f7cdfb4b640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd806eab0 con 0x7f7cd80a9d70 2024-03-29T11:19:56.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.973+0000 7f7cdfb4b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.954 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.973+0000 7f7cdfb4b640 10 monclient(hunting): authenticate will time out at 1409.170717s 2024-03-29T11:19:56.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.978+0000 7f7cde0e8640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7cd8141860 0x7f7cd8137080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:47973/0 (socket says 172.21.15.157:47973) 2024-03-29T11:19:56.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.978+0000 7f7cde0e8640 1 -- 172.21.15.157:0/2291130612 learned_addr learned my addr 172.21.15.157:0/2291130612 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:19:56.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.978+0000 7f7cce7fc640 10 client.?.objecter ms_handle_connect 0x7f7cd8141860 2024-03-29T11:19:56.955 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.978+0000 7f7cce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.956 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.978+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491954869 0 0) 0x7f7cd806e7d0 con 0x7f7cd8141860 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.978+0000 7f7cce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.978+0000 7f7cce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.978+0000 7f7cce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.978+0000 7f7cce7fc640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.978+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc004eb0 con 0x7f7cd8141860 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.979+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3422757179 0 0) 0x7f7cbc004eb0 con 0x7f7cd8141860 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.979+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cbc005300 con 0x7f7cd8141860 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.979+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7cd0001b30 con 0x7f7cd8141860 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.979+0000 7f7cce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.979+0000 7f7cce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.979+0000 7f7cce7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:56.957 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:56.958 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:56.958 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:56.958 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.958 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:56.958 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:56.958 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:56.958 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:56.958 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:56.958 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.979+0000 7f7cce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:56.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.980+0000 7f7cce7fc640 10 client.?.objecter ms_handle_connect 0x7f7cd813f650 2024-03-29T11:19:56.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.980+0000 7f7cce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.980+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019533126 0 0) 0x7f7cd8137790 con 0x7f7cd813f650 2024-03-29T11:19:56.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.980+0000 7f7cce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.980+0000 7f7cce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.980+0000 7f7cce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.980+0000 7f7cce7fc640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:19:56.962 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.980+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc0058a0 con 0x7f7cd813f650 2024-03-29T11:19:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2928296291 0 0) 0x7f7cbc0058a0 con 0x7f7cd813f650 2024-03-29T11:19:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cd8137790 con 0x7f7cd813f650 2024-03-29T11:19:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3807462856 0 0) 0x7f7cbc005300 con 0x7f7cd8141860 2024-03-29T11:19:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 >> v1:172.21.15.157:6790/0 conn(0x7f7cd80a9d70 legacy=0x7f7cd81423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 >> v1:172.21.15.157:6789/0 conn(0x7f7cd813f650 legacy=0x7f7cd8136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 1 monclient: found mon.b 2024-03-29T11:19:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:56.963 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd806ed90 con 0x7f7cd8141860 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.981+0000 7f7cdfb4b640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cdfb4b640 10 monclient: _renew_subs 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cdfb4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cdfb4b640 1 -- 172.21.15.157:0/2291130612 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd8141e10 con 0x7f7cd8141860 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cdfb4b640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cdfb4b640 10 monclient: _renew_subs 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cdfb4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cdfb4b640 1 -- 172.21.15.157:0/2291130612 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd813ff80 con 0x7f7cd8141860 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cd0002100 con 0x7f7cd8141860 2024-03-29T11:19:56.964 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7ccffff640 4 set_mon_vals no callback set 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f7cd0001b30 con 0x7f7cd8141860 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cce7fc640 10 monclient: dump: 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:19:56.965 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.982+0000 7f7cdfb4b640 1 librados: init done 2024-03-29T11:19:56.966 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.984+0000 7f7cbbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.966 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.984+0000 7f7cbbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.966 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.984+0000 7f7cbbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:56.966 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.984+0000 7f7cbbfff640 1 -- 172.21.15.157:0/2291130612 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd8075f10 con 0x7f7cd8141860 2024-03-29T11:19:56.967 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.988+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f7cd001abb0 con 0x7f7cd8141860 2024-03-29T11:19:56.967 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.988+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f7cd0084940 con 0x7f7cd8141860 2024-03-29T11:19:56.967 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.988+0000 7f7cce7fc640 10 client.4516.objecter ms_dispatch 0x7f7cd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:56.967 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.988+0000 7f7cce7fc640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:56.967 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.988+0000 7f7cce7fc640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:56.967 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.988+0000 7f7cce7fc640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T11:19:56.967 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.989+0000 7f7cdd8e7640 10 monclient: get_auth_request con 0x7f7cbc06e380 auth_method 0 2024-03-29T11:19:56.967 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.989+0000 7f7cce7fc640 10 client.4516.objecter ms_handle_connect 0x7f7cbc06e380 2024-03-29T11:19:56.969 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.992+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7cd0050280 con 0x7f7cd8141860 2024-03-29T11:19:56.969 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.992+0000 7f7cce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.969 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:56.992+0000 7f7cce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.293 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.316+0000 7f7cbbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:57.293 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.316+0000 7f7cbbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:57.293 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.316+0000 7f7cbbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:19:57.293 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.316+0000 7f7cbbfff640 1 -- 172.21.15.157:0/2291130612 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cd813ff80 con 0x7f7cd8141860 2024-03-29T11:19:57.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.318+0000 7f7cce7fc640 1 -- 172.21.15.157:0/2291130612 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f7cd813ff80 con 0x7f7cd8141860 2024-03-29T11:19:57.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.318+0000 7f7cce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:57.295 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.318+0000 7f7cce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:57.296 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:19:57.296 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:57.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.323+0000 7f7cbbfff640 10 librados: watch_flush enter 2024-03-29T11:19:57.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.324+0000 7f7cbbfff640 10 librados: watch_flush exit 2024-03-29T11:19:57.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.324+0000 7f7cbbfff640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.324+0000 7f7cbbfff640 10 client.4516.objecter successfully canceled tick 2024-03-29T11:19:57.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.324+0000 7f7cbbfff640 1 -- 172.21.15.157:0/2291130612 >> v1:172.21.15.157:6816/1747058247 conn(0x7f7cbc06e380 legacy=0x7f7cbc070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.324+0000 7f7cbbfff640 10 monclient: shutdown 2024-03-29T11:19:57.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.324+0000 7f7cbbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.324+0000 7f7cbbfff640 1 -- 172.21.15.157:0/2291130612 >> v1:172.21.15.195:6789/0 conn(0x7f7cd8141860 legacy=0x7f7cd8137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.327+0000 7f7cbbfff640 1 -- 172.21.15.157:0/2291130612 shutdown_connections 2024-03-29T11:19:57.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.327+0000 7f7cbbfff640 1 -- 172.21.15.157:0/2291130612 >> 172.21.15.157:0/2291130612 conn(0x7f7cd8066a00 msgr2=0x7f7cd813af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.327+0000 7f7cbbfff640 1 -- 172.21.15.157:0/2291130612 shutdown_connections 2024-03-29T11:19:57.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.327+0000 7f7cbbfff640 1 -- 172.21.15.157:0/2291130612 wait complete. 2024-03-29T11:19:57.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:19:57.327+0000 7f7cbbfff640 1 librados: shutdown 2024-03-29T11:19:57.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:04.129 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:04.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.306+0000 7fc8cb222640 10 monclient: get_monmap_and_config 2024-03-29T11:20:04.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.306+0000 7fc8cb222640 10 monclient: build_initial_monmap 2024-03-29T11:20:04.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.306+0000 7fc8cb222640 10 monclient: monmap: 2024-03-29T11:20:04.283 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:04.283 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:04.284 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:20:04.307290+0000 2024-03-29T11:20:04.284 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:20:04.307290+0000 2024-03-29T11:20:04.284 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:04.284 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:04.284 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:04.284 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:20:04.284 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:20:04.284 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:04.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.307+0000 7fc8cb222640 1 Processor -- start 2024-03-29T11:20:04.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.307+0000 7fc8cb222640 1 -- start start 2024-03-29T11:20:04.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.307+0000 7fc8cb222640 10 monclient: init 2024-03-29T11:20:04.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.308+0000 7fc8cb222640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:04.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.308+0000 7fc8cb222640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:04.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8cb222640 10 monclient(hunting): picked mon.noname-a con 0x7fc8bc0389e0 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:04.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8cb222640 10 monclient(hunting): picked mon.noname-c con 0x7fc8bc038dc0 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8cb222640 10 monclient(hunting): picked mon.noname-b con 0x7fc8bc030660 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8cb222640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc0c54c0 con 0x7fc8bc0389e0 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8cb222640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc036430 con 0x7fc8bc038dc0 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8cb222640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc039840 con 0x7fc8bc030660 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8cb222640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8cb222640 10 monclient(hunting): authenticate will time out at 1416.505956s 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8c9a1f640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc8bc038dc0 0x7fc8bc0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:56414/0 (socket says 172.21.15.157:56414) 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8c9a1f640 1 -- 172.21.15.157:0/1792544866 learned_addr learned my addr 172.21.15.157:0/1792544866 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8c921e640 1 -- 172.21.15.157:0/1792544866 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809613297 0 0) 0x7fc8bc036430 con 0x7fc8bc038dc0 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8c921e640 10 monclient(hunting): _negotiate 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8c921e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8c921e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:04.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8c921e640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:20:04.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.309+0000 7fc8c921e640 1 -- 172.21.15.157:0/1792544866 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b0003520 con 0x7fc8bc038dc0 2024-03-29T11:20:04.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 1 -- 172.21.15.157:0/1792544866 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2512851137 0 0) 0x7fc8b0003520 con 0x7fc8bc038dc0 2024-03-29T11:20:04.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 1 -- 172.21.15.157:0/1792544866 >> v1:172.21.15.157:6790/0 conn(0x7fc8bc030660 legacy=0x7fc8bc0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:04.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 1 -- 172.21.15.157:0/1792544866 >> v1:172.21.15.157:6789/0 conn(0x7fc8bc0389e0 legacy=0x7fc8bc033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: _finish_hunting 0 2024-03-29T11:20:04.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 1 monclient: found mon.noname-c 2024-03-29T11:20:04.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:04.290 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 1 -- 172.21.15.157:0/1792544866 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc035690 con 0x7fc8bc038dc0 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: _finish_auth 0 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: _check_auth_tickets 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 1 -- 172.21.15.157:0/1792544866 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc8c0001690 con 0x7fc8bc038dc0 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: dump: 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.291 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8cb222640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8cb222640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 1 -- 172.21.15.157:0/1792544866 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8c0001920 con 0x7fc8bc038dc0 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 1 -- 172.21.15.157:0/1792544866 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc8c0001cb0 con 0x7fc8bc038dc0 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:04.292 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8c921e640 10 monclient: dump: 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8cb222640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:04.293 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.310+0000 7fc8cb222640 4 set_mon_vals no callback set 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.311+0000 7fc8cb222640 10 monclient: shutdown 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.311+0000 7fc8cb222640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.311+0000 7fc8cb222640 1 -- 172.21.15.157:0/1792544866 >> v1:172.21.15.195:6789/0 conn(0x7fc8bc038dc0 legacy=0x7fc8bc0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.312+0000 7fc8cb222640 1 -- 172.21.15.157:0/1792544866 shutdown_connections 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.312+0000 7fc8cb222640 1 -- 172.21.15.157:0/1792544866 >> 172.21.15.157:0/1792544866 conn(0x7fc8bc005c00 msgr2=0x7fc8bc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.312+0000 7fc8cb222640 1 -- 172.21.15.157:0/1792544866 shutdown_connections 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.312+0000 7fc8cb222640 1 -- 172.21.15.157:0/1792544866 wait complete. 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.313+0000 7fc8cb222640 10 monclient: build_initial_monmap 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.313+0000 7fc8cb222640 10 monclient: monmap: 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:20:04.294 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:20:04.295 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:04.295 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:04.295 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:04.295 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:20:04.295 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:20:04.295 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:04.299 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.313+0000 7fc8cb222640 1 librados: starting msgr at 2024-03-29T11:20:04.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.313+0000 7fc8cb222640 1 librados: starting objecter 2024-03-29T11:20:04.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.313+0000 7fc8cb222640 1 Processor -- start 2024-03-29T11:20:04.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.313+0000 7fc8cb222640 1 -- start start 2024-03-29T11:20:04.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.313+0000 7fc8cb222640 1 librados: setting wanted keys 2024-03-29T11:20:04.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.313+0000 7fc8cb222640 1 librados: calling monclient init 2024-03-29T11:20:04.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.313+0000 7fc8cb222640 10 monclient: init 2024-03-29T11:20:04.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.314+0000 7fc8cb222640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:04.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.314+0000 7fc8cb222640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:04.300 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.314+0000 7fc8cb222640 10 monclient(hunting): picked mon.noname-b con 0x7fc8bc030660 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8cb222640 10 monclient(hunting): picked mon.noname-c con 0x7fc8bc0389e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8cb222640 10 monclient(hunting): picked mon.noname-a con 0x7fc8bc0c79a0 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8cb222640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc02fc10 con 0x7fc8bc0c79a0 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8cb222640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc02ea20 con 0x7fc8bc030660 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8cb222640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc0c29b0 con 0x7fc8bc0389e0 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8ca220640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc8bc030660 0x7fc8bc02f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:58634/0 (socket says 172.21.15.157:58634) 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8ca220640 1 -- 172.21.15.157:0/3150207000 learned_addr learned my addr 172.21.15.157:0/3150207000 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8cb222640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8baffd640 10 client.?.objecter ms_handle_connect 0x7fc8bc030660 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.316+0000 7fc8cb222640 10 monclient(hunting): authenticate will time out at 1416.513761s 2024-03-29T11:20:04.301 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.317+0000 7fc8baffd640 10 client.?.objecter ms_handle_connect 0x7fc8bc0389e0 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.317+0000 7fc8baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.317+0000 7fc8baffd640 10 client.?.objecter ms_handle_connect 0x7fc8bc0c79a0 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.317+0000 7fc8baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.317+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680386244 0 0) 0x7fc8bc02ea20 con 0x7fc8bc030660 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a8004eb0 con 0x7fc8bc030660 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501990455 0 0) 0x7fc8bc0c29b0 con 0x7fc8bc0389e0 2024-03-29T11:20:04.302 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8bc02ea20 con 0x7fc8bc0389e0 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461187234 0 0) 0x7fc8bc02fc10 con 0x7fc8bc0c79a0 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.318+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8bc0c29b0 con 0x7fc8bc0c79a0 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1557469136 0 0) 0x7fc8bc02ea20 con 0x7fc8bc0389e0 2024-03-29T11:20:04.303 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8bc02fc10 con 0x7fc8bc0389e0 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc8c0001550 con 0x7fc8bc0389e0 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient(hunting): dump: 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:04.304 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3788347225 0 0) 0x7fc8bc0c29b0 con 0x7fc8bc0c79a0 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8bc02ea20 con 0x7fc8bc0c79a0 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1310378011 0 0) 0x7fc8a8004eb0 con 0x7fc8bc030660 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8bc0c29b0 con 0x7fc8bc030660 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc8c404fdd0 con 0x7fc8bc0c79a0 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient(hunting): dump: 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.305 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc8b4001990 con 0x7fc8bc030660 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient(hunting): dump: 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:04.306 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3361182485 0 0) 0x7fc8bc02fc10 con 0x7fc8bc0389e0 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 >> v1:172.21.15.195:6789/0 conn(0x7fc8bc030660 legacy=0x7fc8bc02f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 >> v1:172.21.15.157:6789/0 conn(0x7fc8bc0c79a0 legacy=0x7fc8bc15cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 1 monclient: found mon.c 2024-03-29T11:20:04.307 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.319+0000 7fc8baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc0c2b40 con 0x7fc8bc0389e0 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8baffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8cb222640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8cb222640 10 monclient: _renew_subs 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8cb222640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8cb222640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8bc15d410 con 0x7fc8bc0389e0 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8cb222640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8cb222640 10 monclient: _renew_subs 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8cb222640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:04.308 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8cb222640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8bc15d8a0 con 0x7fc8bc0389e0 2024-03-29T11:20:04.309 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.320+0000 7fc8cb222640 1 librados: init done 2024-03-29T11:20:04.309 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.323+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8c0001790 con 0x7fc8bc0389e0 2024-03-29T11:20:04.309 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.323+0000 7fc8baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:04.309 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.323+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc8c0004c30 con 0x7fc8bc0389e0 2024-03-29T11:20:04.309 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.323+0000 7fc8c8a1d640 4 set_mon_vals no callback set 2024-03-29T11:20:04.310 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.323+0000 7fc8b8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.325+0000 7fc8b8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.325+0000 7fc8b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.325+0000 7fc8b8ff9640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ac005080 con 0x7fc8bc0389e0 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.325+0000 7fc8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.325+0000 7fc8baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.325+0000 7fc8baffd640 10 monclient: dump: 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:04.311 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.326+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fc8c001ac40 con 0x7fc8bc0389e0 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.327+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fc8c0084b90 con 0x7fc8bc0389e0 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.327+0000 7fc8baffd640 10 client.4556.objecter ms_dispatch 0x7fc8c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.327+0000 7fc8baffd640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.327+0000 7fc8ca220640 10 monclient: get_auth_request con 0x7fc8a806e190 auth_method 0 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.327+0000 7fc8baffd640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.327+0000 7fc8baffd640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:20:04.312 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.327+0000 7fc8baffd640 10 client.4556.objecter ms_handle_connect 0x7fc8a806e190 2024-03-29T11:20:04.313 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.336+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8c00504f0 con 0x7fc8bc0389e0 2024-03-29T11:20:04.313 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.336+0000 7fc8baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:04.313 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.336+0000 7fc8baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:04.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.681+0000 7fc8b8ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:04.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.681+0000 7fc8b8ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:04.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.681+0000 7fc8b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:04.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.681+0000 7fc8b8ff9640 1 -- 172.21.15.157:0/3150207000 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8ac005330 con 0x7fc8bc0389e0 2024-03-29T11:20:04.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.684+0000 7fc8baffd640 1 -- 172.21.15.157:0/3150207000 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fc8c0052420 con 0x7fc8bc0389e0 2024-03-29T11:20:04.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.684+0000 7fc8baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:04.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.684+0000 7fc8baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:04.662 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:20:04.662 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:04.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.688+0000 7fc8b8ff9640 10 librados: watch_flush enter 2024-03-29T11:20:04.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.688+0000 7fc8b8ff9640 10 librados: watch_flush exit 2024-03-29T11:20:04.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.688+0000 7fc8b8ff9640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:20:04.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.688+0000 7fc8b8ff9640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:20:04.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.690+0000 7fc8b8ff9640 1 -- 172.21.15.157:0/3150207000 >> v1:172.21.15.157:6816/1747058247 conn(0x7fc8a806e190 legacy=0x7fc8a8070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.690+0000 7fc8b8ff9640 10 monclient: shutdown 2024-03-29T11:20:04.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.690+0000 7fc8b8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:04.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.690+0000 7fc8b8ff9640 1 -- 172.21.15.157:0/3150207000 >> v1:172.21.15.157:6790/0 conn(0x7fc8bc0389e0 legacy=0x7fc8bc0c22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.691+0000 7fc8b8ff9640 1 -- 172.21.15.157:0/3150207000 shutdown_connections 2024-03-29T11:20:04.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.691+0000 7fc8b8ff9640 1 -- 172.21.15.157:0/3150207000 >> 172.21.15.157:0/3150207000 conn(0x7fc8bc005c00 msgr2=0x7fc8bc034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:04.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.691+0000 7fc8b8ff9640 1 -- 172.21.15.157:0/3150207000 shutdown_connections 2024-03-29T11:20:04.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.691+0000 7fc8b8ff9640 1 -- 172.21.15.157:0/3150207000 wait complete. 2024-03-29T11:20:04.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:04.692+0000 7fc8b8ff9640 1 librados: shutdown 2024-03-29T11:20:04.689 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:11.496 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:11.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.674+0000 7fc7026ff640 10 monclient: get_monmap_and_config 2024-03-29T11:20:11.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.674+0000 7fc7026ff640 10 monclient: build_initial_monmap 2024-03-29T11:20:11.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.674+0000 7fc7026ff640 10 monclient: monmap: 2024-03-29T11:20:11.651 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:11.651 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:11.651 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:20:11.675338+0000 2024-03-29T11:20:11.651 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:20:11.675338+0000 2024-03-29T11:20:11.651 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:11.651 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:11.652 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:11.652 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:20:11.652 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:20:11.652 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:11.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.675+0000 7fc7026ff640 1 Processor -- start 2024-03-29T11:20:11.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.675+0000 7fc7026ff640 1 -- start start 2024-03-29T11:20:11.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.675+0000 7fc7026ff640 10 monclient: init 2024-03-29T11:20:11.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc7026ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:11.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc7026ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:11.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc7026ff640 10 monclient(hunting): picked mon.noname-a con 0x7fc6fc1457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:11.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc7026ff640 10 monclient(hunting): picked mon.noname-b con 0x7fc6fc13f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:11.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc7026ff640 10 monclient(hunting): picked mon.noname-c con 0x7fc6fc0a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:11.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc7026ff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc0aaef0 con 0x7fc6fc1457f0 2024-03-29T11:20:11.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc7026ff640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc145bd0 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc7026ff640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc145eb0 con 0x7fc6fc13f650 2024-03-29T11:20:11.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc7026ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:11.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc7026ff640 10 monclient(hunting): authenticate will time out at 1423.873161s 2024-03-29T11:20:11.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc6fbfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc6fc1457f0 0x7fc6fc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:59590/0 (socket says 172.21.15.157:59590) 2024-03-29T11:20:11.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc6fbfff640 1 -- 172.21.15.157:0/2393259412 learned_addr learned my addr 172.21.15.157:0/2393259412 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:11.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207555046 0 0) 0x7fc6fc145bd0 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc6faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc6faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc6faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.656 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc6faffd640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.676+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e4003520 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653423458 0 0) 0x7fc6fc0aaef0 con 0x7fc6fc1457f0 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6fc145bd0 con 0x7fc6fc1457f0 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4173174869 0 0) 0x7fc6e4003520 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 >> v1:172.21.15.157:6790/0 conn(0x7fc6fc13f650 legacy=0x7fc6fc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 >> v1:172.21.15.157:6789/0 conn(0x7fc6fc1457f0 legacy=0x7fc6fc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 1 monclient: found mon.noname-c 2024-03-29T11:20:11.657 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc146340 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc6f4001a10 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: dump: 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:11.658 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc7026ff640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc7026ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6f4001b70 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 1 -- 172.21.15.157:0/2393259412 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc6f4001f80 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc6faffd640 10 monclient: dump: 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:11.659 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc7026ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.677+0000 7fc7026ff640 4 set_mon_vals no callback set 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.678+0000 7fc7026ff640 10 monclient: shutdown 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.678+0000 7fc7026ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.678+0000 7fc7026ff640 1 -- 172.21.15.157:0/2393259412 >> v1:172.21.15.195:6789/0 conn(0x7fc6fc0a9d70 legacy=0x7fc6fc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.678+0000 7fc7026ff640 1 -- 172.21.15.157:0/2393259412 shutdown_connections 2024-03-29T11:20:11.660 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.678+0000 7fc7026ff640 1 -- 172.21.15.157:0/2393259412 >> 172.21.15.157:0/2393259412 conn(0x7fc6fc066a00 msgr2=0x7fc6fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.678+0000 7fc7026ff640 1 -- 172.21.15.157:0/2393259412 shutdown_connections 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.678+0000 7fc7026ff640 1 -- 172.21.15.157:0/2393259412 wait complete. 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.679+0000 7fc7026ff640 10 monclient: build_initial_monmap 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.679+0000 7fc7026ff640 10 monclient: monmap: 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:20:11.661 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.680+0000 7fc7026ff640 1 librados: starting msgr at 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.680+0000 7fc7026ff640 1 librados: starting objecter 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.680+0000 7fc7026ff640 1 Processor -- start 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.680+0000 7fc7026ff640 1 -- start start 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.680+0000 7fc7026ff640 1 librados: setting wanted keys 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.680+0000 7fc7026ff640 1 librados: calling monclient init 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.680+0000 7fc7026ff640 10 monclient: init 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.681+0000 7fc7026ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.681+0000 7fc7026ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.681+0000 7fc7026ff640 10 monclient(hunting): picked mon.noname-c con 0x7fc6fc0a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.681+0000 7fc7026ff640 10 monclient(hunting): picked mon.noname-a con 0x7fc6fc13f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:11.664 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.681+0000 7fc7026ff640 10 monclient(hunting): picked mon.noname-b con 0x7fc6fc1457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:11.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.681+0000 7fc7026ff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc06f970 con 0x7fc6fc13f650 2024-03-29T11:20:11.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.681+0000 7fc7026ff640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc13ffb0 con 0x7fc6fc1457f0 2024-03-29T11:20:11.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.681+0000 7fc700c9c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fc6fc1457f0 0x7fc6fc1e6060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:39500/0 (socket says 172.21.15.157:39500) 2024-03-29T11:20:11.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.681+0000 7fc700c9c640 1 -- 172.21.15.157:0/23344483 learned_addr learned my addr 172.21.15.157:0/23344483 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:11.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc06e9c0 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc7026ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:11.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc7026ff640 10 monclient(hunting): authenticate will time out at 1423.878990s 2024-03-29T11:20:11.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc6f8ff9640 10 client.?.objecter ms_handle_connect 0x7fc6fc1457f0 2024-03-29T11:20:11.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc6f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:11.665 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc6f8ff9640 10 client.?.objecter ms_handle_connect 0x7fc6fc13f650 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc6f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316638034 0 0) 0x7fc6fc13ffb0 con 0x7fc6fc1457f0 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc6f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc6f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc6f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.682+0000 7fc6f8ff9640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e8004eb0 con 0x7fc6fc1457f0 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307019378 0 0) 0x7fc6fc06f970 con 0x7fc6fc13f650 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6fc13ffb0 con 0x7fc6fc13f650 2024-03-29T11:20:11.666 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 client.?.objecter ms_handle_connect 0x7fc6fc0a9d70 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2902339210 0 0) 0x7fc6e8004eb0 con 0x7fc6fc1457f0 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6fc06f970 con 0x7fc6fc1457f0 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc6f4001f40 con 0x7fc6fc1457f0 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:11.667 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124303182 0 0) 0x7fc6fc06e9c0 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e8004eb0 con 0x7fc6fc0a9d70 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2771091010 0 0) 0x7fc6fc13ffb0 con 0x7fc6fc13f650 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6fc06e9c0 con 0x7fc6fc13f650 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc6f0001510 con 0x7fc6fc13f650 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:11.668 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.683+0000 7fc6f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 119403680 0 0) 0x7fc6fc06e9c0 con 0x7fc6fc13f650 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 >> v1:172.21.15.157:6790/0 conn(0x7fc6fc0a9d70 legacy=0x7fc6fc1404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 >> v1:172.21.15.195:6789/0 conn(0x7fc6fc1457f0 legacy=0x7fc6fc1e6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:11.669 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 1 monclient: found mon.a 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc06eb50 con 0x7fc6fc13f650 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc6f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc7026ff640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc7026ff640 10 monclient: _renew_subs 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc7026ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6fc06df20 con 0x7fc6fc13f650 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc7026ff640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc7026ff640 10 monclient: _renew_subs 2024-03-29T11:20:11.670 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc7026ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:20:11.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6fc142a30 con 0x7fc6fc13f650 2024-03-29T11:20:11.671 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.684+0000 7fc7026ff640 1 librados: init done 2024-03-29T11:20:11.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.687+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6f0001a20 con 0x7fc6fc13f650 2024-03-29T11:20:11.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.687+0000 7fc6f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:11.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.687+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fc6f0001d70 con 0x7fc6fc13f650 2024-03-29T11:20:11.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.687+0000 7fc6f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.687+0000 7fc6f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:11.672 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.687+0000 7fc6f8ff9640 10 monclient: dump: 2024-03-29T11:20:11.672 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:11.672 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:11.672 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:11.672 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.687+0000 7fc7026ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.687+0000 7fc6fa7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.688+0000 7fc7026ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.688+0000 7fc7026ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.688+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6fc06e2c0 con 0x7fc6fc13f650 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.689+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fc6f0019780 con 0x7fc6fc13f650 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.690+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fc6f0083800 con 0x7fc6fc13f650 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.690+0000 7fc6f8ff9640 10 client.4536.objecter ms_dispatch 0x7fc6fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:11.673 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.690+0000 7fc6f8ff9640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:11.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.690+0000 7fc6f8ff9640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:11.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.690+0000 7fc6fbfff640 10 monclient: get_auth_request con 0x7fc6e806dea0 auth_method 0 2024-03-29T11:20:11.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.690+0000 7fc6f8ff9640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-29T11:20:11.674 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.691+0000 7fc6f8ff9640 10 client.4536.objecter ms_handle_connect 0x7fc6e806dea0 2024-03-29T11:20:11.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.704+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6f004f0e0 con 0x7fc6fc13f650 2024-03-29T11:20:11.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.704+0000 7fc6f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:11.681 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:11.704+0000 7fc6f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:12.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.116+0000 7fc7026ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:12.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.116+0000 7fc7026ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:12.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.116+0000 7fc7026ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:20:12.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.116+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6fc13fda0 con 0x7fc6fc13f650 2024-03-29T11:20:12.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.118+0000 7fc6f8ff9640 1 -- 172.21.15.157:0/23344483 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fc6f0051010 con 0x7fc6fc13f650 2024-03-29T11:20:12.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.118+0000 7fc6f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:12.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.118+0000 7fc6f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:12.095 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:20:12.095 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:12.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 10 librados: watch_flush enter 2024-03-29T11:20:12.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 10 librados: watch_flush exit 2024-03-29T11:20:12.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-29T11:20:12.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 10 client.4536.objecter successfully canceled tick 2024-03-29T11:20:12.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 >> v1:172.21.15.157:6816/1747058247 conn(0x7fc6e806dea0 legacy=0x7fc6e8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 10 monclient: shutdown 2024-03-29T11:20:12.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:12.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 >> v1:172.21.15.157:6789/0 conn(0x7fc6fc13f650 legacy=0x7fc6fc140be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 shutdown_connections 2024-03-29T11:20:12.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 >> 172.21.15.157:0/23344483 conn(0x7fc6fc066a00 msgr2=0x7fc6fc0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:12.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.122+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 shutdown_connections 2024-03-29T11:20:12.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.123+0000 7fc7026ff640 1 -- 172.21.15.157:0/23344483 wait complete. 2024-03-29T11:20:12.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:12.123+0000 7fc7026ff640 1 librados: shutdown 2024-03-29T11:20:12.118 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:18.925 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:19.077 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.100+0000 7ff7fd955640 10 monclient: get_monmap_and_config 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.100+0000 7ff7fd955640 10 monclient: build_initial_monmap 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.100+0000 7ff7fd955640 10 monclient: monmap: 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:20:19.101213+0000 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:20:19.101213+0000 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:20:19.078 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:19.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.101+0000 7ff7fd955640 1 Processor -- start 2024-03-29T11:20:19.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.101+0000 7ff7fd955640 1 -- start start 2024-03-29T11:20:19.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.101+0000 7ff7fd955640 10 monclient: init 2024-03-29T11:20:19.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.102+0000 7ff7fd955640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:19.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.102+0000 7ff7fd955640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:19.079 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.102+0000 7ff7fd955640 10 monclient(hunting): picked mon.noname-a con 0x7ff7f80a9ac0 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:19.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.102+0000 7ff7fd955640 10 monclient(hunting): picked mon.noname-b con 0x7ff7f813ee90 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:19.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.102+0000 7ff7fd955640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f80af900 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:19.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.102+0000 7ff7fd955640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f80afd10 con 0x7ff7f80a9ac0 2024-03-29T11:20:19.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.102+0000 7ff7fd955640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f80af6f0 con 0x7ff7f80af900 2024-03-29T11:20:19.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.102+0000 7ff7fd955640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f80aac80 con 0x7ff7f813ee90 2024-03-29T11:20:19.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.102+0000 7ff7fd955640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:19.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.102+0000 7ff7fd955640 10 monclient(hunting): authenticate will time out at 1431.299272s 2024-03-29T11:20:19.081 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.105+0000 7ff7f77fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff7f80af900 0x7ff7f8149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:37322/0 (socket says 172.21.15.157:37322) 2024-03-29T11:20:19.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.105+0000 7ff7f77fe640 1 -- 172.21.15.157:0/1267007440 learned_addr learned my addr 172.21.15.157:0/1267007440 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:19.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.105+0000 7ff7f5ffb640 1 -- 172.21.15.157:0/1267007440 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046809907 0 0) 0x7ff7f80aac80 con 0x7ff7f813ee90 2024-03-29T11:20:19.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.105+0000 7ff7f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:19.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.105+0000 7ff7f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:19.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.105+0000 7ff7f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:19.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.105+0000 7ff7f5ffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:20:19.082 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.105+0000 7ff7f5ffb640 1 -- 172.21.15.157:0/1267007440 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e0003520 con 0x7ff7f813ee90 2024-03-29T11:20:19.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 1 -- 172.21.15.157:0/1267007440 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 410111716 0 0) 0x7ff7e0003520 con 0x7ff7f813ee90 2024-03-29T11:20:19.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 1 -- 172.21.15.157:0/1267007440 >> v1:172.21.15.195:6789/0 conn(0x7ff7f80af900 legacy=0x7ff7f8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 1 -- 172.21.15.157:0/1267007440 >> v1:172.21.15.157:6789/0 conn(0x7ff7f80a9ac0 legacy=0x7ff7f80adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:19.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:19.083 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 1 -- 172.21.15.157:0/1267007440 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f80aaf60 con 0x7ff7f813ee90 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 1 -- 172.21.15.157:0/1267007440 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff7e8001670 con 0x7ff7f813ee90 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 10 monclient: dump: 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:19.084 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7fd955640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7fd955640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 1 -- 172.21.15.157:0/1267007440 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7e8001900 con 0x7ff7f813ee90 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7fd955640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:19.085 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.106+0000 7ff7fd955640 4 set_mon_vals no callback set 2024-03-29T11:20:19.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.107+0000 7ff7fd955640 10 monclient: shutdown 2024-03-29T11:20:19.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.107+0000 7ff7fd955640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:19.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.107+0000 7ff7fd955640 1 -- 172.21.15.157:0/1267007440 >> v1:172.21.15.157:6790/0 conn(0x7ff7f813ee90 legacy=0x7ff7f813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.107+0000 7ff7fd955640 1 -- 172.21.15.157:0/1267007440 shutdown_connections 2024-03-29T11:20:19.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.107+0000 7ff7fd955640 1 -- 172.21.15.157:0/1267007440 >> 172.21.15.157:0/1267007440 conn(0x7ff7f80b35e0 msgr2=0x7ff7f80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:19.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.109+0000 7ff7fd955640 1 -- 172.21.15.157:0/1267007440 shutdown_connections 2024-03-29T11:20:19.086 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.109+0000 7ff7fd955640 1 -- 172.21.15.157:0/1267007440 wait complete. 2024-03-29T11:20:19.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.110+0000 7ff7fd955640 10 monclient: build_initial_monmap 2024-03-29T11:20:19.087 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.110+0000 7ff7fd955640 10 monclient: monmap: 2024-03-29T11:20:19.087 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:19.087 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:19.087 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:20:19.087 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:20:19.087 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:19.088 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:19.088 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:19.088 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:20:19.088 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:20:19.088 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:19.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.111+0000 7ff7fd955640 1 librados: starting msgr at 2024-03-29T11:20:19.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.111+0000 7ff7fd955640 1 librados: starting objecter 2024-03-29T11:20:19.088 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.111+0000 7ff7fd955640 1 Processor -- start 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.111+0000 7ff7fd955640 1 -- start start 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.111+0000 7ff7fd955640 1 librados: setting wanted keys 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.111+0000 7ff7fd955640 1 librados: calling monclient init 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.111+0000 7ff7fd955640 10 monclient: init 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.112+0000 7ff7fd955640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.112+0000 7ff7fd955640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.112+0000 7ff7fd955640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f80a9ac0 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.112+0000 7ff7fd955640 10 monclient(hunting): picked mon.noname-a con 0x7ff7f80af900 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.112+0000 7ff7fd955640 10 monclient(hunting): picked mon.noname-b con 0x7ff7f813ee90 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.112+0000 7ff7fd955640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f8071480 con 0x7ff7f80af900 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.112+0000 7ff7fd955640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f80716f0 con 0x7ff7f813ee90 2024-03-29T11:20:19.089 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.112+0000 7ff7fd955640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f8070170 con 0x7ff7f80a9ac0 2024-03-29T11:20:19.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.112+0000 7ff7fd955640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:19.090 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.112+0000 7ff7fd955640 10 monclient(hunting): authenticate will time out at 1431.309253s 2024-03-29T11:20:19.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.114+0000 7ff7f77fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff7f813ee90 0x7ff7f81efb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:37630/0 (socket says 172.21.15.157:37630) 2024-03-29T11:20:19.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.114+0000 7ff7f77fe640 1 -- 172.21.15.157:0/791672012 learned_addr learned my addr 172.21.15.157:0/791672012 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:19.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.114+0000 7ff7dffff640 10 client.?.objecter ms_handle_connect 0x7ff7f813ee90 2024-03-29T11:20:19.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.114+0000 7ff7dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:19.092 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.114+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423801754 0 0) 0x7ff7f80716f0 con 0x7ff7f813ee90 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.114+0000 7ff7dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.114+0000 7ff7dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.114+0000 7ff7dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.114+0000 7ff7dffff640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.114+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e4004eb0 con 0x7ff7f813ee90 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.115+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3213622713 0 0) 0x7ff7e4004eb0 con 0x7ff7f813ee90 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.115+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7f80716f0 con 0x7ff7f813ee90 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.115+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff7ec0018b0 con 0x7ff7f813ee90 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.115+0000 7ff7dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.115+0000 7ff7dffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.115+0000 7ff7dffff640 10 monclient(hunting): dump: 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:19.093 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:19.094 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.094 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:19.094 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:19.094 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:19.094 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:19.094 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:19.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.115+0000 7ff7dffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:19.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.115+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4140014054 0 0) 0x7ff7f80716f0 con 0x7ff7f813ee90 2024-03-29T11:20:19.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.115+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 >> v1:172.21.15.157:6790/0 conn(0x7ff7f80a9ac0 legacy=0x7ff7f81e42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.094 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.115+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 >> v1:172.21.15.157:6789/0 conn(0x7ff7f80af900 legacy=0x7ff7f81e13f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7dffff640 1 monclient: found mon.b 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7dffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f806f370 con 0x7ff7f813ee90 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7dffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7fd955640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7fd955640 10 monclient: _renew_subs 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7fd955640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:19.095 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7fd955640 1 -- 172.21.15.157:0/791672012 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f8070690 con 0x7ff7f813ee90 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7fd955640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7fd955640 10 monclient: _renew_subs 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7fd955640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7fd955640 1 -- 172.21.15.157:0/791672012 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f81e4bd0 con 0x7ff7f813ee90 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.117+0000 7ff7fd955640 1 librados: init done 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.118+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7ec001a10 con 0x7ff7f813ee90 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.118+0000 7ff7dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.118+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff7ec004a80 con 0x7ff7f813ee90 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.118+0000 7ff7dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.118+0000 7ff7dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.118+0000 7ff7dffff640 10 monclient: dump: 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.096 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:19.097 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:19.097 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:19.097 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:19.097 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:19.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.118+0000 7ff7f4ff9640 4 set_mon_vals no callback set 2024-03-29T11:20:19.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.118+0000 7ff7fd955640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:19.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.119+0000 7ff7fd955640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:19.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.119+0000 7ff7fd955640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:19.097 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.119+0000 7ff7fd955640 1 -- 172.21.15.157:0/791672012 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d8005080 con 0x7ff7f813ee90 2024-03-29T11:20:19.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.121+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7ff7ec004cc0 con 0x7ff7f813ee90 2024-03-29T11:20:19.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.122+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7ff7ec084ad0 con 0x7ff7f813ee90 2024-03-29T11:20:19.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.122+0000 7ff7dffff640 10 client.4534.objecter ms_dispatch 0x7ff7f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:19.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.122+0000 7ff7dffff640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:19.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.122+0000 7ff7dffff640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:19.099 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.122+0000 7ff7dffff640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-29T11:20:19.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.123+0000 7ff7f6ffd640 10 monclient: get_auth_request con 0x7ff7e406dc70 auth_method 0 2024-03-29T11:20:19.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.123+0000 7ff7dffff640 10 client.4534.objecter ms_handle_connect 0x7ff7e406dc70 2024-03-29T11:20:19.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.128+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7ec0503b0 con 0x7ff7f813ee90 2024-03-29T11:20:19.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.128+0000 7ff7dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:19.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.128+0000 7ff7dffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:19.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.468+0000 7ff7fd955640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:19.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.468+0000 7ff7fd955640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:19.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.468+0000 7ff7fd955640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:19.445 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.468+0000 7ff7fd955640 1 -- 172.21.15.157:0/791672012 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7d8005330 con 0x7ff7f813ee90 2024-03-29T11:20:19.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.469+0000 7ff7dffff640 1 -- 172.21.15.157:0/791672012 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7ff7ec0522e0 con 0x7ff7f813ee90 2024-03-29T11:20:19.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.469+0000 7ff7dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:19.446 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.469+0000 7ff7dffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:19.447 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:20:19.447 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:19.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.473+0000 7ff7fd955640 10 librados: watch_flush enter 2024-03-29T11:20:19.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.473+0000 7ff7fd955640 10 librados: watch_flush exit 2024-03-29T11:20:19.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.473+0000 7ff7fd955640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-29T11:20:19.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.473+0000 7ff7fd955640 10 client.4534.objecter successfully canceled tick 2024-03-29T11:20:19.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.473+0000 7ff7fd955640 1 -- 172.21.15.157:0/791672012 >> v1:172.21.15.157:6816/1747058247 conn(0x7ff7e406dc70 legacy=0x7ff7e4070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.473+0000 7ff7fd955640 10 monclient: shutdown 2024-03-29T11:20:19.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.473+0000 7ff7fd955640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:19.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.474+0000 7ff7fd955640 1 -- 172.21.15.157:0/791672012 >> v1:172.21.15.195:6789/0 conn(0x7ff7f813ee90 legacy=0x7ff7f81efb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.474+0000 7ff7fd955640 1 -- 172.21.15.157:0/791672012 shutdown_connections 2024-03-29T11:20:19.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.474+0000 7ff7fd955640 1 -- 172.21.15.157:0/791672012 >> 172.21.15.157:0/791672012 conn(0x7ff7f80b35e0 msgr2=0x7ff7f80ad1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:19.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.474+0000 7ff7fd955640 1 -- 172.21.15.157:0/791672012 shutdown_connections 2024-03-29T11:20:19.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.474+0000 7ff7fd955640 1 -- 172.21.15.157:0/791672012 wait complete. 2024-03-29T11:20:19.452 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:19.474+0000 7ff7fd955640 1 librados: shutdown 2024-03-29T11:20:19.468 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:26.275 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:26.423 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.445+0000 7ff57e1bc640 10 monclient: get_monmap_and_config 2024-03-29T11:20:26.423 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.446+0000 7ff57e1bc640 10 monclient: build_initial_monmap 2024-03-29T11:20:26.423 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.446+0000 7ff57e1bc640 10 monclient: monmap: 2024-03-29T11:20:26.424 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:26.424 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:26.424 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:20:26.447125+0000 2024-03-29T11:20:26.424 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:20:26.447125+0000 2024-03-29T11:20:26.424 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:26.424 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:26.424 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:26.424 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:20:26.424 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:20:26.424 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:26.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.447+0000 7ff57e1bc640 1 Processor -- start 2024-03-29T11:20:26.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.448+0000 7ff57e1bc640 1 -- start start 2024-03-29T11:20:26.425 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.448+0000 7ff57e1bc640 10 monclient: init 2024-03-29T11:20:26.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff57e1bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:26.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff57e1bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:26.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff57e1bc640 10 monclient(hunting): picked mon.noname-c con 0x7ff5780a9750 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:26.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff57e1bc640 10 monclient(hunting): picked mon.noname-a con 0x7ff57813e1e0 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:26.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff57e1bc640 10 monclient(hunting): picked mon.noname-b con 0x7ff578071c30 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:26.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff57e1bc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5780ab300 con 0x7ff57813e1e0 2024-03-29T11:20:26.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff57e1bc640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5780ac2f0 con 0x7ff5780a9750 2024-03-29T11:20:26.426 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff57e1bc640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5780ad8d0 con 0x7ff578071c30 2024-03-29T11:20:26.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff57e1bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:26.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff57e1bc640 10 monclient(hunting): authenticate will time out at 1438.646425s 2024-03-29T11:20:26.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff5777fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff5780a9750 0x7ff57813dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:60938/0 (socket says 172.21.15.157:60938) 2024-03-29T11:20:26.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff5777fe640 1 -- 172.21.15.157:0/1158730727 learned_addr learned my addr 172.21.15.157:0/1158730727 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:26.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff5767fc640 1 -- 172.21.15.157:0/1158730727 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826930942 0 0) 0x7ff5780ac2f0 con 0x7ff5780a9750 2024-03-29T11:20:26.427 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff5767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:26.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff5767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:26.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff5767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:26.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.449+0000 7ff5767fc640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:20:26.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.450+0000 7ff5767fc640 1 -- 172.21.15.157:0/1158730727 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff560003520 con 0x7ff5780a9750 2024-03-29T11:20:26.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.450+0000 7ff5767fc640 1 -- 172.21.15.157:0/1158730727 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3905816218 0 0) 0x7ff560003520 con 0x7ff5780a9750 2024-03-29T11:20:26.428 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.450+0000 7ff5767fc640 1 -- 172.21.15.157:0/1158730727 >> v1:172.21.15.157:6790/0 conn(0x7ff578071c30 legacy=0x7ff5780ace20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:26.429 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 1 -- 172.21.15.157:0/1158730727 >> v1:172.21.15.157:6789/0 conn(0x7ff57813e1e0 legacy=0x7ff5780ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.429 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:26.429 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 1 monclient: found mon.noname-c 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 1 -- 172.21.15.157:0/1158730727 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5780ada60 con 0x7ff5780a9750 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 1 -- 172.21.15.157:0/1158730727 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff568001690 con 0x7ff5780a9750 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.452+0000 7ff5767fc640 10 monclient: dump: 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:26.430 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.453+0000 7ff57e1bc640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.453+0000 7ff57e1bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.453+0000 7ff5767fc640 1 -- 172.21.15.157:0/1158730727 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff568001860 con 0x7ff5780a9750 2024-03-29T11:20:26.431 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.453+0000 7ff5767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.453+0000 7ff5767fc640 1 -- 172.21.15.157:0/1158730727 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff568001cb0 con 0x7ff5780a9750 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.453+0000 7ff5767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.453+0000 7ff5767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.453+0000 7ff5767fc640 10 monclient: dump: 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:26.432 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:26.433 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:26.433 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.453+0000 7ff57e1bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:26.433 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.453+0000 7ff57e1bc640 4 set_mon_vals no callback set 2024-03-29T11:20:26.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.454+0000 7ff57e1bc640 10 monclient: shutdown 2024-03-29T11:20:26.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.455+0000 7ff57e1bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:26.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.455+0000 7ff57e1bc640 1 -- 172.21.15.157:0/1158730727 >> v1:172.21.15.195:6789/0 conn(0x7ff5780a9750 legacy=0x7ff57813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.455+0000 7ff57e1bc640 1 -- 172.21.15.157:0/1158730727 shutdown_connections 2024-03-29T11:20:26.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.455+0000 7ff57e1bc640 1 -- 172.21.15.157:0/1158730727 >> 172.21.15.157:0/1158730727 conn(0x7ff57806b660 msgr2=0x7ff57806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:26.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.456+0000 7ff57e1bc640 1 -- 172.21.15.157:0/1158730727 shutdown_connections 2024-03-29T11:20:26.434 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.456+0000 7ff57e1bc640 1 -- 172.21.15.157:0/1158730727 wait complete. 2024-03-29T11:20:26.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.458+0000 7ff57e1bc640 10 monclient: build_initial_monmap 2024-03-29T11:20:26.435 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.458+0000 7ff57e1bc640 10 monclient: monmap: 2024-03-29T11:20:26.435 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:26.435 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:26.435 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:20:26.435 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:20:26.436 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:26.436 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:26.436 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:26.436 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:20:26.436 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:20:26.436 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:26.436 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.458+0000 7ff57e1bc640 1 librados: starting msgr at 2024-03-29T11:20:26.436 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.458+0000 7ff57e1bc640 1 librados: starting objecter 2024-03-29T11:20:26.436 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.459+0000 7ff57e1bc640 1 Processor -- start 2024-03-29T11:20:26.437 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.459+0000 7ff57e1bc640 1 -- start start 2024-03-29T11:20:26.437 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.459+0000 7ff57e1bc640 1 librados: setting wanted keys 2024-03-29T11:20:26.437 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.459+0000 7ff57e1bc640 1 librados: calling monclient init 2024-03-29T11:20:26.437 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.459+0000 7ff57e1bc640 10 monclient: init 2024-03-29T11:20:26.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff57e1bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:26.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff57e1bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:26.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff57e1bc640 10 monclient(hunting): picked mon.noname-b con 0x7ff578071c30 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:26.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff57e1bc640 10 monclient(hunting): picked mon.noname-a con 0x7ff5780a9750 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:26.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff57e1bc640 10 monclient(hunting): picked mon.noname-c con 0x7ff57813e1e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:26.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff57e1bc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff578137760 con 0x7ff5780a9750 2024-03-29T11:20:26.440 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff57e1bc640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff578137a20 con 0x7ff578071c30 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff57e1bc640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff578136d80 con 0x7ff57813e1e0 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff57e1bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff57e1bc640 10 monclient(hunting): authenticate will time out at 1438.658717s 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff5777fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7ff578071c30 0x7ff578135a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:33112/0 (socket says 172.21.15.157:33112) 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.461+0000 7ff5777fe640 1 -- 172.21.15.157:0/991170 learned_addr learned my addr 172.21.15.157:0/991170 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 10 client.?.objecter ms_handle_connect 0x7ff578071c30 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156504646 0 0) 0x7ff578137a20 con 0x7ff578071c30 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:26.441 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff564004eb0 con 0x7ff578071c30 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1441165936 0 0) 0x7ff564004eb0 con 0x7ff578071c30 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff578137a20 con 0x7ff578071c30 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff568001b70 con 0x7ff578071c30 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:26.442 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.443 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:26.443 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:26.443 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:26.443 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:26.443 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:26.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:26.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1673035757 0 0) 0x7ff578137a20 con 0x7ff578071c30 2024-03-29T11:20:26.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 >> v1:172.21.15.157:6790/0 conn(0x7ff57813e1e0 legacy=0x7ff5781dda40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:26.443 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.462+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 >> v1:172.21.15.157:6789/0 conn(0x7ff5780a9750 legacy=0x7ff5781d3000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:26.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:26.447 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 1 monclient: found mon.b 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff578137060 con 0x7ff578071c30 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff57e1bc640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff57e1bc640 10 monclient: _renew_subs 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff57e1bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff57e1bc640 1 -- 172.21.15.157:0/991170 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff578137340 con 0x7ff578071c30 2024-03-29T11:20:26.448 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff568004bf0 con 0x7ff578071c30 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff568004fc0 con 0x7ff578071c30 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff55ffff640 10 monclient: dump: 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:26.449 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff57e1bc640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff57e1bc640 10 monclient: _renew_subs 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.465+0000 7ff57e1bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.466+0000 7ff57e1bc640 1 -- 172.21.15.157:0/991170 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5781d3710 con 0x7ff578071c30 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.466+0000 7ff57e1bc640 1 librados: init done 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.467+0000 7ff575ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.469+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7ff568005120 con 0x7ff578071c30 2024-03-29T11:20:26.450 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.469+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7ff568084d30 con 0x7ff578071c30 2024-03-29T11:20:26.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.469+0000 7ff55ffff640 10 client.4540.objecter ms_dispatch 0x7ff578000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:26.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.469+0000 7ff55ffff640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:26.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.469+0000 7ff55ffff640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:26.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.470+0000 7ff55ffff640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-29T11:20:26.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.470+0000 7ff576ffd640 10 monclient: get_auth_request con 0x7ff56406dd00 auth_method 0 2024-03-29T11:20:26.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.470+0000 7ff55ffff640 10 client.4540.objecter ms_handle_connect 0x7ff56406dd00 2024-03-29T11:20:26.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.470+0000 7ff57e1bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:26.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.470+0000 7ff57e1bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:26.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.471+0000 7ff57e1bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:26.451 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.471+0000 7ff57e1bc640 1 -- 172.21.15.157:0/991170 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff558005080 con 0x7ff578071c30 2024-03-29T11:20:26.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.478+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff568050610 con 0x7ff578071c30 2024-03-29T11:20:26.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.478+0000 7ff55ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:26.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.478+0000 7ff55ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:26.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.802+0000 7ff57e1bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:26.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.802+0000 7ff57e1bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:26.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.802+0000 7ff57e1bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:26.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.802+0000 7ff57e1bc640 1 -- 172.21.15.157:0/991170 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff558005330 con 0x7ff578071c30 2024-03-29T11:20:26.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.803+0000 7ff55ffff640 1 -- 172.21.15.157:0/991170 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7ff568052540 con 0x7ff578071c30 2024-03-29T11:20:26.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.803+0000 7ff55ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:26.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.803+0000 7ff55ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:26.784 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:20:26.784 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:26.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.807+0000 7ff57e1bc640 10 librados: watch_flush enter 2024-03-29T11:20:26.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.808+0000 7ff57e1bc640 10 librados: watch_flush exit 2024-03-29T11:20:26.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.808+0000 7ff57e1bc640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-29T11:20:26.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.808+0000 7ff57e1bc640 10 client.4540.objecter successfully canceled tick 2024-03-29T11:20:26.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.808+0000 7ff57e1bc640 1 -- 172.21.15.157:0/991170 >> v1:172.21.15.157:6816/1747058247 conn(0x7ff56406dd00 legacy=0x7ff5640701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.808+0000 7ff57e1bc640 10 monclient: shutdown 2024-03-29T11:20:26.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.808+0000 7ff57e1bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:26.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.808+0000 7ff57e1bc640 1 -- 172.21.15.157:0/991170 >> v1:172.21.15.195:6789/0 conn(0x7ff578071c30 legacy=0x7ff578135a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.809+0000 7ff57e1bc640 1 -- 172.21.15.157:0/991170 shutdown_connections 2024-03-29T11:20:26.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.809+0000 7ff57e1bc640 1 -- 172.21.15.157:0/991170 >> 172.21.15.157:0/991170 conn(0x7ff57806b660 msgr2=0x7ff5780ae3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:26.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.810+0000 7ff57e1bc640 1 -- 172.21.15.157:0/991170 shutdown_connections 2024-03-29T11:20:26.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.810+0000 7ff57e1bc640 1 -- 172.21.15.157:0/991170 wait complete. 2024-03-29T11:20:26.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:26.810+0000 7ff57e1bc640 1 librados: shutdown 2024-03-29T11:20:26.806 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:33.614 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.786+0000 7fbfbebb5640 10 monclient: get_monmap_and_config 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.786+0000 7fbfbebb5640 10 monclient: build_initial_monmap 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.787+0000 7fbfbebb5640 10 monclient: monmap: 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:20:33.788106+0000 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:20:33.788106+0000 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:33.764 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:20:33.765 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:20:33.765 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:33.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.788+0000 7fbfbebb5640 1 Processor -- start 2024-03-29T11:20:33.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.788+0000 7fbfbebb5640 1 -- start start 2024-03-29T11:20:33.765 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.788+0000 7fbfbebb5640 10 monclient: init 2024-03-29T11:20:33.766 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.790+0000 7fbfbebb5640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:33.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.790+0000 7fbfbebb5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:33.767 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.790+0000 7fbfbebb5640 10 monclient(hunting): picked mon.noname-a con 0x7fbfb80a9ac0 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:33.768 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.791+0000 7fbfbebb5640 10 monclient(hunting): picked mon.noname-b con 0x7fbfb813ee90 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:33.768 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.791+0000 7fbfbebb5640 10 monclient(hunting): picked mon.noname-c con 0x7fbfb80af900 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:33.768 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.791+0000 7fbfbebb5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb80afd10 con 0x7fbfb80a9ac0 2024-03-29T11:20:33.768 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.791+0000 7fbfbebb5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb80af6f0 con 0x7fbfb80af900 2024-03-29T11:20:33.768 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.791+0000 7fbfbebb5640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb80aac80 con 0x7fbfb813ee90 2024-03-29T11:20:33.768 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.791+0000 7fbfbebb5640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:33.768 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.791+0000 7fbfbebb5640 10 monclient(hunting): authenticate will time out at 1445.988427s 2024-03-29T11:20:33.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.793+0000 7fbfbd152640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbfb80af900 0x7fbfb8149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:43602/0 (socket says 172.21.15.157:43602) 2024-03-29T11:20:33.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.793+0000 7fbfbd152640 1 -- 172.21.15.157:0/593617342 learned_addr learned my addr 172.21.15.157:0/593617342 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:33.770 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963545022 0 0) 0x7fbfb80af6f0 con 0x7fbfb80af900 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa8003520 con 0x7fbfb80af900 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351134066 0 0) 0x7fbfb80aac80 con 0x7fbfb813ee90 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:20:33.771 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfb80af6f0 con 0x7fbfb813ee90 2024-03-29T11:20:33.772 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1067574515 0 0) 0x7fbfb80af6f0 con 0x7fbfb813ee90 2024-03-29T11:20:33.772 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 >> v1:172.21.15.195:6789/0 conn(0x7fbfb80af900 legacy=0x7fbfb8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.772 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 >> v1:172.21.15.157:6789/0 conn(0x7fbfb80a9ac0 legacy=0x7fbfb80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:33.772 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:33.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 1 monclient: found mon.noname-b 2024-03-29T11:20:33.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:33.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:20:33.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb80aaf60 con 0x7fbfb813ee90 2024-03-29T11:20:33.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:33.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:33.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:33.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fbfa4001670 con 0x7fbfb813ee90 2024-03-29T11:20:33.773 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfb77fe640 10 monclient: dump: 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfbebb5640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:20:33.774 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.794+0000 7fbfbebb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfa40017d0 con 0x7fbfb813ee90 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfbebb5640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfb77fe640 1 -- 172.21.15.157:0/593617342 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fbfa4001c20 con 0x7fbfb813ee90 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfbebb5640 4 set_mon_vals no callback set 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfb77fe640 10 monclient: dump: 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.775 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:33.776 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:33.776 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:33.776 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:33.776 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:33.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfbebb5640 10 monclient: shutdown 2024-03-29T11:20:33.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfbebb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:33.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfbebb5640 1 -- 172.21.15.157:0/593617342 >> v1:172.21.15.157:6790/0 conn(0x7fbfb813ee90 legacy=0x7fbfb813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfbebb5640 1 -- 172.21.15.157:0/593617342 shutdown_connections 2024-03-29T11:20:33.776 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.795+0000 7fbfbebb5640 1 -- 172.21.15.157:0/593617342 >> 172.21.15.157:0/593617342 conn(0x7fbfb80b35e0 msgr2=0x7fbfb80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:33.778 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.796+0000 7fbfbebb5640 1 -- 172.21.15.157:0/593617342 shutdown_connections 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.796+0000 7fbfbebb5640 1 -- 172.21.15.157:0/593617342 wait complete. 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.797+0000 7fbfbebb5640 10 monclient: build_initial_monmap 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.797+0000 7fbfbebb5640 10 monclient: monmap: 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:20:33.779 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.798+0000 7fbfbebb5640 1 librados: starting msgr at 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.798+0000 7fbfbebb5640 1 librados: starting objecter 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.798+0000 7fbfbebb5640 1 Processor -- start 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.798+0000 7fbfbebb5640 1 -- start start 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.799+0000 7fbfbebb5640 1 librados: setting wanted keys 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.799+0000 7fbfbebb5640 1 librados: calling monclient init 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.799+0000 7fbfbebb5640 10 monclient: init 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.800+0000 7fbfbebb5640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.800+0000 7fbfbebb5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.800+0000 7fbfbebb5640 10 monclient(hunting): picked mon.noname-c con 0x7fbfb80af900 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.800+0000 7fbfbebb5640 10 monclient(hunting): picked mon.noname-a con 0x7fbfb813ee90 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:33.780 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.800+0000 7fbfbebb5640 10 monclient(hunting): picked mon.noname-b con 0x7fbfb8120cc0 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:33.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.800+0000 7fbfbebb5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb81210a0 con 0x7fbfb813ee90 2024-03-29T11:20:33.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.800+0000 7fbfbebb5640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb812d100 con 0x7fbfb8120cc0 2024-03-29T11:20:33.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.800+0000 7fbfbebb5640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb811be70 con 0x7fbfb80af900 2024-03-29T11:20:33.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.800+0000 7fbfbebb5640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:33.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.800+0000 7fbfbebb5640 10 monclient(hunting): authenticate will time out at 1445.997833s 2024-03-29T11:20:33.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.801+0000 7fbfbd152640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbfb8120cc0 0x7fbfb806eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:43712/0 (socket says 172.21.15.157:43712) 2024-03-29T11:20:33.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.801+0000 7fbfbd152640 1 -- 172.21.15.157:0/3859952129 learned_addr learned my addr 172.21.15.157:0/3859952129 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:33.781 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.802+0000 7fbfb57fa640 10 client.?.objecter ms_handle_connect 0x7fbfb8120cc0 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.802+0000 7fbfb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198054322 0 0) 0x7fbfb812d100 con 0x7fbfb8120cc0 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9c004eb0 con 0x7fbfb8120cc0 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1295033590 0 0) 0x7fbf9c004eb0 con 0x7fbfb8120cc0 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfb812d100 con 0x7fbfb8120cc0 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fbfb0001cd0 con 0x7fbfb8120cc0 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.782 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 10 monclient(hunting): dump: 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.803+0000 7fbfb57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:33.783 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.804+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4120827657 0 0) 0x7fbfb812d100 con 0x7fbfb8120cc0 2024-03-29T11:20:33.784 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.804+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 >> v1:172.21.15.157:6790/0 conn(0x7fbfb80af900 legacy=0x7fbfb8071170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:33.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 >> v1:172.21.15.157:6789/0 conn(0x7fbfb813ee90 legacy=0x7fbfb806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.785 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 1 monclient: found mon.b 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb8072250 con 0x7fbfb8120cc0 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 10 client.?.objecter ms_handle_connect 0x7fbfb813ee90 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfbebb5640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfbebb5640 10 monclient: _renew_subs 2024-03-29T11:20:33.786 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfbebb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:33.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfbebb5640 1 -- 172.21.15.157:0/3859952129 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb8071bc0 con 0x7fbfb8120cc0 2024-03-29T11:20:33.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfbebb5640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:33.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfbebb5640 10 monclient: _renew_subs 2024-03-29T11:20:33.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfbebb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:33.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfbebb5640 1 -- 172.21.15.157:0/3859952129 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb806f5c0 con 0x7fbfb8120cc0 2024-03-29T11:20:33.787 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.807+0000 7fbfbebb5640 1 librados: init done 2024-03-29T11:20:33.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.809+0000 7fbfbebb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.809+0000 7fbfbebb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.809+0000 7fbfbebb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:33.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.809+0000 7fbfbebb5640 1 -- 172.21.15.157:0/3859952129 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb80720a0 con 0x7fbfb8120cc0 2024-03-29T11:20:33.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.811+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfb0001fd0 con 0x7fbfb8120cc0 2024-03-29T11:20:33.788 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.811+0000 7fbfb57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:33.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.811+0000 7fbfb6ffd640 4 set_mon_vals no callback set 2024-03-29T11:20:33.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.812+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fbfb0004fb0 con 0x7fbfb8120cc0 2024-03-29T11:20:33.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.812+0000 7fbfb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.812+0000 7fbfb57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:33.790 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.812+0000 7fbfb57fa640 10 monclient: dump: 2024-03-29T11:20:33.790 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:33.790 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:33.790 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:33.790 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.812+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fbfb00051d0 con 0x7fbfb8120cc0 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.813+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fbfb0084f70 con 0x7fbfb8120cc0 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.813+0000 7fbfb57fa640 10 client.4546.objecter ms_dispatch 0x7fbfb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.813+0000 7fbfb57fa640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.813+0000 7fbfb57fa640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.813+0000 7fbfbc951640 10 monclient: get_auth_request con 0x7fbf9c06db80 auth_method 0 2024-03-29T11:20:33.791 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.813+0000 7fbfb57fa640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:20:33.792 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.816+0000 7fbfb57fa640 10 client.4546.objecter ms_handle_connect 0x7fbf9c06db80 2024-03-29T11:20:33.799 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.819+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbfb00508d0 con 0x7fbfb8120cc0 2024-03-29T11:20:33.799 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.820+0000 7fbfb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.799 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:33.820+0000 7fbfb57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:34.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.140+0000 7fbfbebb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:34.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.141+0000 7fbfbebb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:34.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.141+0000 7fbfbebb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:20:34.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.141+0000 7fbfbebb5640 1 -- 172.21.15.157:0/3859952129 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfb80723e0 con 0x7fbfb8120cc0 2024-03-29T11:20:34.120 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:20:34.120 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:34.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.142+0000 7fbfb57fa640 1 -- 172.21.15.157:0/3859952129 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fbfb0052800 con 0x7fbfb8120cc0 2024-03-29T11:20:34.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.142+0000 7fbfb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:34.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.142+0000 7fbfb57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.145+0000 7fbfbebb5640 10 librados: watch_flush enter 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.145+0000 7fbfbebb5640 10 librados: watch_flush exit 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.145+0000 7fbfbebb5640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.145+0000 7fbfbebb5640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.146+0000 7fbfbebb5640 1 -- 172.21.15.157:0/3859952129 >> v1:172.21.15.157:6816/1747058247 conn(0x7fbf9c06db80 legacy=0x7fbf9c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.146+0000 7fbfbebb5640 10 monclient: shutdown 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.146+0000 7fbfbebb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.146+0000 7fbfbebb5640 1 -- 172.21.15.157:0/3859952129 >> v1:172.21.15.195:6789/0 conn(0x7fbfb8120cc0 legacy=0x7fbfb806eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.146+0000 7fbfbebb5640 1 -- 172.21.15.157:0/3859952129 shutdown_connections 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.146+0000 7fbfbebb5640 1 -- 172.21.15.157:0/3859952129 >> 172.21.15.157:0/3859952129 conn(0x7fbfb80b35e0 msgr2=0x7fbfb8130150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.146+0000 7fbfbebb5640 1 -- 172.21.15.157:0/3859952129 shutdown_connections 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.146+0000 7fbfbebb5640 1 -- 172.21.15.157:0/3859952129 wait complete. 2024-03-29T11:20:34.123 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:34.146+0000 7fbfbebb5640 1 librados: shutdown 2024-03-29T11:20:34.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:40.945 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:41.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.121+0000 7fdc498e6640 10 monclient: get_monmap_and_config 2024-03-29T11:20:41.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.121+0000 7fdc498e6640 10 monclient: build_initial_monmap 2024-03-29T11:20:41.098 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.121+0000 7fdc498e6640 10 monclient: monmap: 2024-03-29T11:20:41.098 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:41.099 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:41.099 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:20:41.122774+0000 2024-03-29T11:20:41.099 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:20:41.122774+0000 2024-03-29T11:20:41.099 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:41.099 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:41.099 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:41.099 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:20:41.099 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:20:41.099 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:41.100 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.124+0000 7fdc498e6640 1 Processor -- start 2024-03-29T11:20:41.101 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.124+0000 7fdc498e6640 1 -- start start 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.124+0000 7fdc498e6640 10 monclient: init 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.125+0000 7fdc498e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.125+0000 7fdc498e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.125+0000 7fdc498e6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc3c0389e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.125+0000 7fdc498e6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc3c038dc0 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.125+0000 7fdc498e6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc3c030660 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.125+0000 7fdc498e6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c0c54c0 con 0x7fdc3c038dc0 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.125+0000 7fdc498e6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c036430 con 0x7fdc3c030660 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.125+0000 7fdc498e6640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c039840 con 0x7fdc3c0389e0 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.125+0000 7fdc498e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:41.102 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.125+0000 7fdc498e6640 10 monclient(hunting): authenticate will time out at 1453.322013s 2024-03-29T11:20:41.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.126+0000 7fdc42ffd640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fdc3c0389e0 0x7fdc3c033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:39622/0 (socket says 172.21.15.157:39622) 2024-03-29T11:20:41.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.126+0000 7fdc42ffd640 1 -- 172.21.15.157:0/4226671669 learned_addr learned my addr 172.21.15.157:0/4226671669 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:41.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.128+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184644721 0 0) 0x7fdc3c039840 con 0x7fdc3c0389e0 2024-03-29T11:20:41.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.128+0000 7fdc41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:41.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.128+0000 7fdc41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:41.105 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.128+0000 7fdc41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:41.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.128+0000 7fdc41ffb640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:20:41.106 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.128+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc2c003520 con 0x7fdc3c0389e0 2024-03-29T11:20:41.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.129+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970651394 0 0) 0x7fdc3c0c54c0 con 0x7fdc3c038dc0 2024-03-29T11:20:41.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.130+0000 7fdc41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:41.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.130+0000 7fdc41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:41.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.130+0000 7fdc41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:41.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.130+0000 7fdc41ffb640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:20:41.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.130+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc3c039840 con 0x7fdc3c038dc0 2024-03-29T11:20:41.107 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.130+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3558363937 0 0) 0x7fdc3c039840 con 0x7fdc3c038dc0 2024-03-29T11:20:41.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 >> v1:172.21.15.157:6790/0 conn(0x7fdc3c0389e0 legacy=0x7fdc3c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 >> v1:172.21.15.195:6789/0 conn(0x7fdc3c030660 legacy=0x7fdc3c0d3170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:20:41.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:41.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 1 monclient: found mon.noname-a 2024-03-29T11:20:41.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:41.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:20:41.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c035690 con 0x7fdc3c038dc0 2024-03-29T11:20:41.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:41.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:41.110 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdc38001690 con 0x7fdc3c038dc0 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.131+0000 7fdc41ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.132+0000 7fdc41ffb640 10 monclient: dump: 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:41.111 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.132+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc38001920 con 0x7fdc3c038dc0 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.132+0000 7fdc41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.132+0000 7fdc41ffb640 1 -- 172.21.15.157:0/4226671669 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdc38001cb0 con 0x7fdc3c038dc0 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.132+0000 7fdc41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.132+0000 7fdc41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.132+0000 7fdc41ffb640 10 monclient: dump: 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:41.112 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:41.113 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:41.113 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:41.113 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.132+0000 7fdc498e6640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:20:41.113 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.132+0000 7fdc498e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:41.113 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.132+0000 7fdc498e6640 4 set_mon_vals no callback set 2024-03-29T11:20:41.113 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.133+0000 7fdc498e6640 10 monclient: shutdown 2024-03-29T11:20:41.113 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.133+0000 7fdc498e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:41.114 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.133+0000 7fdc498e6640 1 -- 172.21.15.157:0/4226671669 >> v1:172.21.15.157:6789/0 conn(0x7fdc3c038dc0 legacy=0x7fdc3c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.114 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.135+0000 7fdc498e6640 1 -- 172.21.15.157:0/4226671669 shutdown_connections 2024-03-29T11:20:41.114 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.135+0000 7fdc498e6640 1 -- 172.21.15.157:0/4226671669 >> 172.21.15.157:0/4226671669 conn(0x7fdc3c005c00 msgr2=0x7fdc3c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.135+0000 7fdc498e6640 1 -- 172.21.15.157:0/4226671669 shutdown_connections 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.135+0000 7fdc498e6640 1 -- 172.21.15.157:0/4226671669 wait complete. 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.136+0000 7fdc498e6640 10 monclient: build_initial_monmap 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.137+0000 7fdc498e6640 10 monclient: monmap: 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:20:41.115 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:41.116 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.137+0000 7fdc498e6640 1 librados: starting msgr at 2024-03-29T11:20:41.116 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.137+0000 7fdc498e6640 1 librados: starting objecter 2024-03-29T11:20:41.116 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.137+0000 7fdc498e6640 1 Processor -- start 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.138+0000 7fdc498e6640 1 -- start start 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.138+0000 7fdc498e6640 1 librados: setting wanted keys 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.138+0000 7fdc498e6640 1 librados: calling monclient init 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.138+0000 7fdc498e6640 10 monclient: init 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.139+0000 7fdc498e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.139+0000 7fdc498e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.139+0000 7fdc498e6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc3c030660 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.139+0000 7fdc498e6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc3c0389e0 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.139+0000 7fdc498e6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc3c02f940 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.139+0000 7fdc498e6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c0c6d40 con 0x7fdc3c030660 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.139+0000 7fdc498e6640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c0c6fb0 con 0x7fdc3c0389e0 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.139+0000 7fdc498e6640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c0c7740 con 0x7fdc3c02f940 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.139+0000 7fdc498e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:41.117 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.139+0000 7fdc498e6640 10 monclient(hunting): authenticate will time out at 1453.336745s 2024-03-29T11:20:41.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.141+0000 7fdc437fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fdc3c02f940 0x7fdc3c168120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:43878/0 (socket says 172.21.15.157:43878) 2024-03-29T11:20:41.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.141+0000 7fdc437fe640 1 -- 172.21.15.157:0/2747221343 learned_addr learned my addr 172.21.15.157:0/2747221343 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:41.118 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.142+0000 7fdc2bfff640 10 client.?.objecter ms_handle_connect 0x7fdc3c02f940 2024-03-29T11:20:41.119 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.142+0000 7fdc2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:41.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.143+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118422103 0 0) 0x7fdc3c0c7740 con 0x7fdc3c02f940 2024-03-29T11:20:41.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.143+0000 7fdc2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:41.121 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.143+0000 7fdc2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:41.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.143+0000 7fdc2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:41.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.143+0000 7fdc2bfff640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:20:41.122 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.143+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc30004eb0 con 0x7fdc3c02f940 2024-03-29T11:20:41.124 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.145+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 831772235 0 0) 0x7fdc30004eb0 con 0x7fdc3c02f940 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.145+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc3c0c7740 con 0x7fdc3c02f940 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.145+0000 7fdc2bfff640 10 client.?.objecter ms_handle_connect 0x7fdc3c030660 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.145+0000 7fdc2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.145+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdc340017f0 con 0x7fdc3c02f940 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.145+0000 7fdc2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.145+0000 7fdc2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.145+0000 7fdc2bfff640 10 monclient(hunting): dump: 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:41.125 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325994765 0 0) 0x7fdc3c0c6d40 con 0x7fdc3c030660 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc30004eb0 con 0x7fdc3c030660 2024-03-29T11:20:41.126 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1292244973 0 0) 0x7fdc3c0c7740 con 0x7fdc3c02f940 2024-03-29T11:20:41.127 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 >> v1:172.21.15.195:6789/0 conn(0x7fdc3c0389e0 legacy=0x7fdc3c0c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.127 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 >> v1:172.21.15.157:6789/0 conn(0x7fdc3c030660 legacy=0x7fdc3c160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.127 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.146+0000 7fdc2bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:41.127 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc2bfff640 1 monclient: found mon.c 2024-03-29T11:20:41.127 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:41.127 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:41.127 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c02ff70 con 0x7fdc3c02f940 2024-03-29T11:20:41.127 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc2bfff640 10 monclient: _finish_auth 0 2024-03-29T11:20:41.127 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc2bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:41.127 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:41.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc498e6640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:20:41.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc498e6640 10 monclient: _renew_subs 2024-03-29T11:20:41.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc498e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:41.129 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc498e6640 1 -- 172.21.15.157:0/2747221343 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c0c5b10 con 0x7fdc3c02f940 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc498e6640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc498e6640 10 monclient: _renew_subs 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc498e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc498e6640 1 -- 172.21.15.157:0/2747221343 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c0c8310 con 0x7fdc3c02f940 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.147+0000 7fdc498e6640 1 librados: init done 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.149+0000 7fdc498e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.149+0000 7fdc498e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.149+0000 7fdc498e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.149+0000 7fdc498e6640 1 -- 172.21.15.157:0/2747221343 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c030210 con 0x7fdc3c02f940 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.151+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc34001dc0 con 0x7fdc3c02f940 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.151+0000 7fdc2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:41.130 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.151+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fdc34004a40 con 0x7fdc3c02f940 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.151+0000 7fdc2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.151+0000 7fdc2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.151+0000 7fdc2bfff640 10 monclient: dump: 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:41.131 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:41.132 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.153+0000 7fdc40ff9640 4 set_mon_vals no callback set 2024-03-29T11:20:41.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.154+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fdc3401aa10 con 0x7fdc3c02f940 2024-03-29T11:20:41.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.155+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fdc340849e0 con 0x7fdc3c02f940 2024-03-29T11:20:41.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.155+0000 7fdc2bfff640 10 client.4571.objecter ms_dispatch 0x7fdc44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:41.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.155+0000 7fdc2bfff640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:41.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.155+0000 7fdc2bfff640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:41.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.155+0000 7fdc2bfff640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:20:41.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.155+0000 7fdc42ffd640 10 monclient: get_auth_request con 0x7fdc3006d9b0 auth_method 0 2024-03-29T11:20:41.133 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.155+0000 7fdc2bfff640 10 client.4571.objecter ms_handle_connect 0x7fdc3006d9b0 2024-03-29T11:20:41.140 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.163+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc340500f0 con 0x7fdc3c02f940 2024-03-29T11:20:41.140 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.163+0000 7fdc2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:41.140 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.163+0000 7fdc2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:41.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.487+0000 7fdc498e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:41.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.487+0000 7fdc498e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:41.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.487+0000 7fdc498e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:41.464 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.487+0000 7fdc498e6640 1 -- 172.21.15.157:0/2747221343 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc3c007b90 con 0x7fdc3c02f940 2024-03-29T11:20:41.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.488+0000 7fdc2bfff640 1 -- 172.21.15.157:0/2747221343 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fdc340521f0 con 0x7fdc3c02f940 2024-03-29T11:20:41.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.488+0000 7fdc2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:41.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.488+0000 7fdc2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:41.465 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:20:41.465 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:41.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.491+0000 7fdc498e6640 10 librados: watch_flush enter 2024-03-29T11:20:41.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.492+0000 7fdc498e6640 10 librados: watch_flush exit 2024-03-29T11:20:41.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.492+0000 7fdc498e6640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:20:41.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.492+0000 7fdc498e6640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:20:41.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.492+0000 7fdc498e6640 1 -- 172.21.15.157:0/2747221343 >> v1:172.21.15.157:6816/1747058247 conn(0x7fdc3006d9b0 legacy=0x7fdc3006fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.492+0000 7fdc498e6640 10 monclient: shutdown 2024-03-29T11:20:41.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.492+0000 7fdc498e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:41.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.492+0000 7fdc498e6640 1 -- 172.21.15.157:0/2747221343 >> v1:172.21.15.157:6790/0 conn(0x7fdc3c02f940 legacy=0x7fdc3c168120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.492+0000 7fdc498e6640 1 -- 172.21.15.157:0/2747221343 shutdown_connections 2024-03-29T11:20:41.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.492+0000 7fdc498e6640 1 -- 172.21.15.157:0/2747221343 >> 172.21.15.157:0/2747221343 conn(0x7fdc3c005c00 msgr2=0x7fdc3c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:41.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.493+0000 7fdc498e6640 1 -- 172.21.15.157:0/2747221343 shutdown_connections 2024-03-29T11:20:41.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.493+0000 7fdc498e6640 1 -- 172.21.15.157:0/2747221343 wait complete. 2024-03-29T11:20:41.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:41.493+0000 7fdc498e6640 1 librados: shutdown 2024-03-29T11:20:41.487 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:48.295 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:48.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.472+0000 7ff551f2e640 10 monclient: get_monmap_and_config 2024-03-29T11:20:48.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.472+0000 7ff551f2e640 10 monclient: build_initial_monmap 2024-03-29T11:20:48.449 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.472+0000 7ff551f2e640 10 monclient: monmap: 2024-03-29T11:20:48.449 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:48.449 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:48.449 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:20:48.473500+0000 2024-03-29T11:20:48.450 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:20:48.473500+0000 2024-03-29T11:20:48.450 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:48.450 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:48.450 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:48.450 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:20:48.450 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:20:48.450 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:48.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.475+0000 7ff551f2e640 1 Processor -- start 2024-03-29T11:20:48.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.477+0000 7ff551f2e640 1 -- start start 2024-03-29T11:20:48.453 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.477+0000 7ff551f2e640 10 monclient: init 2024-03-29T11:20:48.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.478+0000 7ff551f2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:48.454 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.478+0000 7ff551f2e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:48.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.478+0000 7ff551f2e640 10 monclient(hunting): picked mon.noname-a con 0x7ff54c0a9750 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:48.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.478+0000 7ff551f2e640 10 monclient(hunting): picked mon.noname-c con 0x7ff54c0add40 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:48.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.478+0000 7ff551f2e640 10 monclient(hunting): picked mon.noname-b con 0x7ff54c0aad20 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:48.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.478+0000 7ff551f2e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c149870 con 0x7ff54c0a9750 2024-03-29T11:20:48.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.478+0000 7ff551f2e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c0abf80 con 0x7ff54c0add40 2024-03-29T11:20:48.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.478+0000 7ff551f2e640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c149ac0 con 0x7ff54c0aad20 2024-03-29T11:20:48.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.478+0000 7ff551f2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:48.455 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.478+0000 7ff551f2e640 10 monclient(hunting): authenticate will time out at 1460.675859s 2024-03-29T11:20:48.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.479+0000 7ff55172d640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ff54c0aad20 0x7ff54c151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:55186/0 (socket says 172.21.15.157:55186) 2024-03-29T11:20:48.456 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.479+0000 7ff55172d640 1 -- 172.21.15.157:0/107733391 learned_addr learned my addr 172.21.15.157:0/107733391 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:48.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 1 -- 172.21.15.157:0/107733391 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995568268 0 0) 0x7ff54c149ac0 con 0x7ff54c0aad20 2024-03-29T11:20:48.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:20:48.457 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 1 -- 172.21.15.157:0/107733391 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff538003520 con 0x7ff54c0aad20 2024-03-29T11:20:48.460 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 1 -- 172.21.15.157:0/107733391 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2080148603 0 0) 0x7ff538003520 con 0x7ff54c0aad20 2024-03-29T11:20:48.460 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 1 -- 172.21.15.157:0/107733391 >> v1:172.21.15.195:6789/0 conn(0x7ff54c0add40 legacy=0x7ff54c143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.460 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 1 -- 172.21.15.157:0/107733391 >> v1:172.21.15.157:6789/0 conn(0x7ff54c0a9750 legacy=0x7ff54c149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.460 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:48.460 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 1 -- 172.21.15.157:0/107733391 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c14a6f0 con 0x7ff54c0aad20 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 1 -- 172.21.15.157:0/107733391 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff544001a10 con 0x7ff54c0aad20 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff54b7fe640 10 monclient: dump: 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:48.461 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff551f2e640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.481+0000 7ff551f2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.482+0000 7ff54b7fe640 1 -- 172.21.15.157:0/107733391 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff544001b70 con 0x7ff54c0aad20 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.482+0000 7ff54b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.482+0000 7ff54b7fe640 1 -- 172.21.15.157:0/107733391 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff544001fc0 con 0x7ff54c0aad20 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.482+0000 7ff54b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.482+0000 7ff54b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:48.462 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.482+0000 7ff54b7fe640 10 monclient: dump: 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.482+0000 7ff551f2e640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:48.463 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.482+0000 7ff551f2e640 4 set_mon_vals no callback set 2024-03-29T11:20:48.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.483+0000 7ff551f2e640 10 monclient: shutdown 2024-03-29T11:20:48.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.483+0000 7ff551f2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:48.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.483+0000 7ff551f2e640 1 -- 172.21.15.157:0/107733391 >> v1:172.21.15.157:6790/0 conn(0x7ff54c0aad20 legacy=0x7ff54c151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.483+0000 7ff551f2e640 1 -- 172.21.15.157:0/107733391 shutdown_connections 2024-03-29T11:20:48.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.483+0000 7ff551f2e640 1 -- 172.21.15.157:0/107733391 >> 172.21.15.157:0/107733391 conn(0x7ff54c0673e0 msgr2=0x7ff54c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:48.465 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.484+0000 7ff551f2e640 1 -- 172.21.15.157:0/107733391 shutdown_connections 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.484+0000 7ff551f2e640 1 -- 172.21.15.157:0/107733391 wait complete. 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.485+0000 7ff551f2e640 10 monclient: build_initial_monmap 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.485+0000 7ff551f2e640 10 monclient: monmap: 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:48.466 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.485+0000 7ff551f2e640 1 librados: starting msgr at 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.485+0000 7ff551f2e640 1 librados: starting objecter 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.486+0000 7ff551f2e640 1 Processor -- start 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.486+0000 7ff551f2e640 1 -- start start 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.486+0000 7ff551f2e640 1 librados: setting wanted keys 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.486+0000 7ff551f2e640 1 librados: calling monclient init 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.486+0000 7ff551f2e640 10 monclient: init 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.487+0000 7ff551f2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.487+0000 7ff551f2e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.487+0000 7ff551f2e640 10 monclient(hunting): picked mon.noname-c con 0x7ff54c0a9750 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.487+0000 7ff550f2c640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7ff54c0a9750 0x7ff54c070aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:57580/0 (socket says 172.21.15.157:57580) 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.488+0000 7ff550f2c640 1 -- 172.21.15.157:0/1851373942 learned_addr learned my addr 172.21.15.157:0/1851373942 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.488+0000 7ff551f2e640 10 monclient(hunting): picked mon.noname-a con 0x7ff54c0aad20 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.488+0000 7ff551f2e640 10 monclient(hunting): picked mon.noname-b con 0x7ff54c0add40 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.488+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c0721f0 con 0x7ff54c0aad20 2024-03-29T11:20:48.467 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.488+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c0724d0 con 0x7ff54c0add40 2024-03-29T11:20:48.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.488+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c13be20 con 0x7ff54c0a9750 2024-03-29T11:20:48.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.488+0000 7ff551f2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:48.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.488+0000 7ff551f2e640 10 monclient(hunting): authenticate will time out at 1460.685759s 2024-03-29T11:20:48.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.490+0000 7ff5497fa640 10 client.?.objecter ms_handle_connect 0x7ff54c0a9750 2024-03-29T11:20:48.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.490+0000 7ff5497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:48.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236437468 0 0) 0x7ff54c13be20 con 0x7ff54c0a9750 2024-03-29T11:20:48.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.468 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:20:48.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff52c004eb0 con 0x7ff54c0a9750 2024-03-29T11:20:48.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 client.?.objecter ms_handle_connect 0x7ff54c0aad20 2024-03-29T11:20:48.469 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284343301 0 0) 0x7ff54c0721f0 con 0x7ff54c0aad20 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff52c0051d0 con 0x7ff54c0aad20 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2323109264 0 0) 0x7ff52c004eb0 con 0x7ff54c0a9750 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff52c005640 con 0x7ff54c0a9750 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff5340019f0 con 0x7ff54c0a9750 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): dump: 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:48.470 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.492+0000 7ff5497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.493+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1515621629 0 0) 0x7ff52c005640 con 0x7ff54c0a9750 2024-03-29T11:20:48.471 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.493+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 >> v1:172.21.15.195:6789/0 conn(0x7ff54c0add40 legacy=0x7ff54c1db450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:48.473 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.494+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 >> v1:172.21.15.157:6789/0 conn(0x7ff54c0aad20 legacy=0x7ff54c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.494+0000 7ff5497fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.494+0000 7ff5497fa640 1 monclient: found mon.c 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.494+0000 7ff5497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.494+0000 7ff5497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.494+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c0711b0 con 0x7ff54c0a9750 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.494+0000 7ff5497fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.494+0000 7ff5497fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.494+0000 7ff5497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.495+0000 7ff551f2e640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.495+0000 7ff551f2e640 10 monclient: _renew_subs 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.495+0000 7ff551f2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.495+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54c13b7e0 con 0x7ff54c0a9750 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.495+0000 7ff551f2e640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:48.474 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.495+0000 7ff551f2e640 10 monclient: _renew_subs 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.495+0000 7ff551f2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.495+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c1dbb60 con 0x7ff54c0a9750 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.495+0000 7ff551f2e640 1 librados: init done 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.496+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5340035f0 con 0x7ff54c0a9750 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.496+0000 7ff5497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.496+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7ff5340019f0 con 0x7ff54c0a9750 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.497+0000 7ff5497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.497+0000 7ff5497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.497+0000 7ff5497fa640 10 monclient: dump: 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.475 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:48.476 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:48.476 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:48.476 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:48.476 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:48.477 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.497+0000 7ff54affd640 4 set_mon_vals no callback set 2024-03-29T11:20:48.477 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.497+0000 7ff551f2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:48.477 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.498+0000 7ff551f2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:48.477 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.498+0000 7ff551f2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:48.477 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.498+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff530005080 con 0x7ff54c0a9750 2024-03-29T11:20:48.478 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.498+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7ff5340038a0 con 0x7ff54c0a9750 2024-03-29T11:20:48.478 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.499+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7ff5340846b0 con 0x7ff54c0a9750 2024-03-29T11:20:48.478 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.499+0000 7ff5497fa640 10 client.4577.objecter ms_dispatch 0x7ff54c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:48.478 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.499+0000 7ff5497fa640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:48.478 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.499+0000 7ff5497fa640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:48.478 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.499+0000 7ff5497fa640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:20:48.478 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.499+0000 7ff54bfff640 10 monclient: get_auth_request con 0x7ff52c06e4d0 auth_method 0 2024-03-29T11:20:48.478 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.500+0000 7ff5497fa640 10 client.4577.objecter ms_handle_connect 0x7ff52c06e4d0 2024-03-29T11:20:48.485 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.509+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff53404fdc0 con 0x7ff54c0a9750 2024-03-29T11:20:48.486 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.509+0000 7ff5497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:48.486 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.509+0000 7ff5497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:48.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.833+0000 7ff551f2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:48.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.833+0000 7ff551f2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:48.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.833+0000 7ff551f2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:48.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.833+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff530005330 con 0x7ff54c0a9750 2024-03-29T11:20:48.812 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.835+0000 7ff5497fa640 1 -- 172.21.15.157:0/1851373942 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7ff53404fdc0 con 0x7ff54c0a9750 2024-03-29T11:20:48.812 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.835+0000 7ff5497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:48.812 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.835+0000 7ff5497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:48.812 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:20:48.812 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:48.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.839+0000 7ff551f2e640 10 librados: watch_flush enter 2024-03-29T11:20:48.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.839+0000 7ff551f2e640 10 librados: watch_flush exit 2024-03-29T11:20:48.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.839+0000 7ff551f2e640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:20:48.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.839+0000 7ff551f2e640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:20:48.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.839+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 >> v1:172.21.15.157:6816/1747058247 conn(0x7ff52c06e4d0 legacy=0x7ff52c070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.839+0000 7ff551f2e640 10 monclient: shutdown 2024-03-29T11:20:48.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.839+0000 7ff551f2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:48.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.839+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 >> v1:172.21.15.157:6790/0 conn(0x7ff54c0a9750 legacy=0x7ff54c070aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.839+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 shutdown_connections 2024-03-29T11:20:48.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.839+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 >> 172.21.15.157:0/1851373942 conn(0x7ff54c0673e0 msgr2=0x7ff54c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:48.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.840+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 shutdown_connections 2024-03-29T11:20:48.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.840+0000 7ff551f2e640 1 -- 172.21.15.157:0/1851373942 wait complete. 2024-03-29T11:20:48.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:48.840+0000 7ff551f2e640 1 librados: shutdown 2024-03-29T11:20:48.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:55.641 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.818+0000 7f9a0e00f640 10 monclient: get_monmap_and_config 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.818+0000 7f9a0e00f640 10 monclient: build_initial_monmap 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.818+0000 7f9a0e00f640 10 monclient: monmap: 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:20:55.819504+0000 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:20:55.819504+0000 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:55.795 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:20:55.796 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:20:55.796 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:55.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.819+0000 7f9a0e00f640 1 Processor -- start 2024-03-29T11:20:55.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.819+0000 7f9a0e00f640 1 -- start start 2024-03-29T11:20:55.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.819+0000 7f9a0e00f640 10 monclient: init 2024-03-29T11:20:55.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.820+0000 7f9a0e00f640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:55.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.820+0000 7f9a0e00f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:55.797 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.820+0000 7f9a0e00f640 10 monclient(hunting): picked mon.noname-b con 0x7f9a080b1f60 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:55.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.820+0000 7f9a0e00f640 10 monclient(hunting): picked mon.noname-c con 0x7f9a08071eb0 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:55.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.820+0000 7f9a0e00f640 10 monclient(hunting): picked mon.noname-a con 0x7f9a08143d10 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:55.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.820+0000 7f9a0e00f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0806ef80 con 0x7f9a08143d10 2024-03-29T11:20:55.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.820+0000 7f9a0e00f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0806f260 con 0x7f9a08071eb0 2024-03-29T11:20:55.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.820+0000 7f9a0e00f640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a080b40c0 con 0x7f9a080b1f60 2024-03-29T11:20:55.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.820+0000 7f9a0e00f640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:55.798 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.820+0000 7f9a0e00f640 10 monclient(hunting): authenticate will time out at 1468.017307s 2024-03-29T11:20:55.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.825+0000 7f9a077fe640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f9a080b1f60 0x7f9a08143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:55357/0 (socket says 172.21.15.157:55357) 2024-03-29T11:20:55.801 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.825+0000 7f9a077fe640 1 -- 172.21.15.157:0/3396349391 learned_addr learned my addr 172.21.15.157:0/3396349391 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:55.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.825+0000 7f9a067fc640 1 -- 172.21.15.157:0/3396349391 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801316101 0 0) 0x7f9a080b40c0 con 0x7f9a080b1f60 2024-03-29T11:20:55.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.825+0000 7f9a067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:55.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.825+0000 7f9a067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:55.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.825+0000 7f9a067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:55.802 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.825+0000 7f9a067fc640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:20:55.803 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.825+0000 7f9a067fc640 1 -- 172.21.15.157:0/3396349391 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f0003520 con 0x7f9a080b1f60 2024-03-29T11:20:55.805 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 1 -- 172.21.15.157:0/3396349391 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973224034 0 0) 0x7f9a0806ef80 con 0x7f9a08143d10 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 1 -- 172.21.15.157:0/3396349391 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a080b40c0 con 0x7f9a08143d10 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 1 -- 172.21.15.157:0/3396349391 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 578494072 0 0) 0x7f9a080b40c0 con 0x7f9a08143d10 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 1 -- 172.21.15.157:0/3396349391 >> v1:172.21.15.157:6790/0 conn(0x7f9a080b1f60 legacy=0x7f9a08143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 1 -- 172.21.15.157:0/3396349391 >> v1:172.21.15.195:6789/0 conn(0x7f9a08071eb0 legacy=0x7f9a08146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 1 monclient: found mon.noname-a 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:20:55.806 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 1 -- 172.21.15.157:0/3396349391 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a08155d70 con 0x7f9a08143d10 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a0e00f640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a0e00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 1 -- 172.21.15.157:0/3396349391 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f99f8001670 con 0x7f9a08143d10 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a067fc640 10 monclient: dump: 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.807 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.826+0000 7f9a0e00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.827+0000 7f9a067fc640 1 -- 172.21.15.157:0/3396349391 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99f80018c0 con 0x7f9a08143d10 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.827+0000 7f9a067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.827+0000 7f9a0e00f640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.827+0000 7f9a0e00f640 4 set_mon_vals no callback set 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.827+0000 7f9a0e00f640 10 monclient: shutdown 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.827+0000 7f9a0e00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.827+0000 7f9a0e00f640 1 -- 172.21.15.157:0/3396349391 >> v1:172.21.15.157:6789/0 conn(0x7f9a08143d10 legacy=0x7f9a08146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.829+0000 7f9a0e00f640 1 -- 172.21.15.157:0/3396349391 shutdown_connections 2024-03-29T11:20:55.808 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.829+0000 7f9a0e00f640 1 -- 172.21.15.157:0/3396349391 >> 172.21.15.157:0/3396349391 conn(0x7f9a08066a00 msgr2=0x7f9a08066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:55.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.833+0000 7f9a0e00f640 1 -- 172.21.15.157:0/3396349391 shutdown_connections 2024-03-29T11:20:55.810 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.833+0000 7f9a0e00f640 1 -- 172.21.15.157:0/3396349391 wait complete. 2024-03-29T11:20:55.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.834+0000 7f9a0e00f640 10 monclient: build_initial_monmap 2024-03-29T11:20:55.811 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.834+0000 7f9a0e00f640 10 monclient: monmap: 2024-03-29T11:20:55.811 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:20:55.811 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:55.811 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:20:55.811 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:20:55.811 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:55.811 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:55.812 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:20:55.812 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:20:55.812 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:20:55.812 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:55.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.835+0000 7f9a0e00f640 1 librados: starting msgr at 2024-03-29T11:20:55.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.835+0000 7f9a0e00f640 1 librados: starting objecter 2024-03-29T11:20:55.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.835+0000 7f9a0e00f640 1 Processor -- start 2024-03-29T11:20:55.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.835+0000 7f9a0e00f640 1 -- start start 2024-03-29T11:20:55.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.835+0000 7f9a0e00f640 1 librados: setting wanted keys 2024-03-29T11:20:55.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.835+0000 7f9a0e00f640 1 librados: calling monclient init 2024-03-29T11:20:55.815 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.835+0000 7f9a0e00f640 10 monclient: init 2024-03-29T11:20:55.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.836+0000 7f9a0e00f640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:55.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.836+0000 7f9a0e00f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:55.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.836+0000 7f9a0e00f640 10 monclient(hunting): picked mon.noname-b con 0x7f9a080b1f60 addr v1:172.21.15.195:6789/0 2024-03-29T11:20:55.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.836+0000 7f9a0e00f640 10 monclient(hunting): picked mon.noname-a con 0x7f9a08143d10 addr v1:172.21.15.157:6789/0 2024-03-29T11:20:55.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.836+0000 7f9a0e00f640 10 monclient(hunting): picked mon.noname-c con 0x7f9a08142b10 addr v1:172.21.15.157:6790/0 2024-03-29T11:20:55.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.836+0000 7f9a0e00f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0806f990 con 0x7f9a08143d10 2024-03-29T11:20:55.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.836+0000 7f9a0e00f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a08132740 con 0x7f9a080b1f60 2024-03-29T11:20:55.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.836+0000 7f9a0e00f640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a08153530 con 0x7f9a08142b10 2024-03-29T11:20:55.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.836+0000 7f9a0e00f640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:55.816 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.836+0000 7f9a0e00f640 10 monclient(hunting): authenticate will time out at 1468.033465s 2024-03-29T11:20:55.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.839+0000 7f9a077fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f9a080b1f60 0x7f9a08142400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:35339/0 (socket says 172.21.15.157:35339) 2024-03-29T11:20:55.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.839+0000 7f9a07fff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f9a08142b10 0x7f9a08132030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:55449/0 (socket says 172.21.15.157:55449) 2024-03-29T11:20:55.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.839+0000 7f9a07fff640 1 -- 172.21.15.157:0/3305793498 learned_addr learned my addr 172.21.15.157:0/3305793498 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:20:55.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.840+0000 7f99effff640 10 client.?.objecter ms_handle_connect 0x7f9a08142b10 2024-03-29T11:20:55.817 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.840+0000 7f99effff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:55.824 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.843+0000 7f99effff640 10 client.?.objecter ms_handle_connect 0x7f9a080b1f60 2024-03-29T11:20:55.824 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.843+0000 7f99effff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:55.824 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.843+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280174448 0 0) 0x7f9a08132740 con 0x7f9a080b1f60 2024-03-29T11:20:55.824 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.843+0000 7f99effff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:55.824 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.843+0000 7f99effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:55.824 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.843+0000 7f99effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:55.824 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:20:55.824 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f4004eb0 con 0x7f9a080b1f60 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983656533 0 0) 0x7f9a08153530 con 0x7f9a08142b10 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a08132740 con 0x7f9a08142b10 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 10 client.?.objecter ms_handle_connect 0x7f9a08143d10 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2522656288 0 0) 0x7f9a08132740 con 0x7f9a08142b10 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.844+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a08153530 con 0x7f9a08142b10 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f99f8001bd0 con 0x7f9a08142b10 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.825 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 10 monclient(hunting): dump: 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1620149608 0 0) 0x7f9a08153530 con 0x7f9a08142b10 2024-03-29T11:20:55.826 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 >> v1:172.21.15.195:6789/0 conn(0x7f9a080b1f60 legacy=0x7f9a08142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 >> v1:172.21.15.157:6789/0 conn(0x7f9a08143d10 legacy=0x7f9a08131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 1 monclient: found mon.c 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a08143060 con 0x7f9a08142b10 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 10 monclient: _finish_auth 0 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.845+0000 7f99effff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f99effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f9a0e00f640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f9a0e00f640 10 monclient: _renew_subs 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f9a0e00f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f9a0e00f640 1 -- 172.21.15.157:0/3305793498 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a08070530 con 0x7f9a08142b10 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99f80038a0 con 0x7f9a08142b10 2024-03-29T11:20:55.827 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f99effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f99f8003c70 con 0x7f9a08142b10 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f99effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f99effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f99effff640 10 monclient: dump: 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:20:55.828 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f9a0e00f640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:55.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f9a0e00f640 10 monclient: _renew_subs 2024-03-29T11:20:55.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f9a0e00f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:55.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f9a0e00f640 1 -- 172.21.15.157:0/3305793498 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a08070af0 con 0x7f9a08142b10 2024-03-29T11:20:55.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f9a0e00f640 1 librados: init done 2024-03-29T11:20:55.829 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.846+0000 7f9a05ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:55.830 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.848+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f99f8003dd0 con 0x7f9a08142b10 2024-03-29T11:20:55.830 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.849+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f99f8084b80 con 0x7f9a08142b10 2024-03-29T11:20:55.830 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.849+0000 7f99effff640 10 client.4583.objecter ms_dispatch 0x7f9a08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:55.830 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.849+0000 7f99effff640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:55.831 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.849+0000 7f99effff640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:55.831 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.849+0000 7f99effff640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T11:20:55.831 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.849+0000 7f9a077fe640 10 monclient: get_auth_request con 0x7f99f406dd00 auth_method 0 2024-03-29T11:20:55.831 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.850+0000 7f99effff640 10 client.4583.objecter ms_handle_connect 0x7f99f406dd00 2024-03-29T11:20:55.831 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.850+0000 7f99edffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:55.831 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.850+0000 7f99edffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:55.831 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.850+0000 7f99edffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:55.831 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.850+0000 7f99edffb640 1 -- 172.21.15.157:0/3305793498 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e0005080 con 0x7f9a08142b10 2024-03-29T11:20:55.837 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.860+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f99f8050460 con 0x7f9a08142b10 2024-03-29T11:20:55.837 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.860+0000 7f99effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:55.837 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:55.860+0000 7f99effff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:56.170 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.193+0000 7f99edffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:56.170 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.193+0000 7f99edffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:56.170 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.193+0000 7f99edffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:20:56.170 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.193+0000 7f99edffb640 1 -- 172.21.15.157:0/3305793498 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99e0005330 con 0x7f9a08142b10 2024-03-29T11:20:56.171 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.195+0000 7f99effff640 1 -- 172.21.15.157:0/3305793498 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f99f8052390 con 0x7f9a08142b10 2024-03-29T11:20:56.172 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.195+0000 7f99effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:56.172 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.195+0000 7f99effff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:56.174 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:20:56.174 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:56.177 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.201+0000 7f99edffb640 10 librados: watch_flush enter 2024-03-29T11:20:56.178 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.201+0000 7f99edffb640 10 librados: watch_flush exit 2024-03-29T11:20:56.178 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.201+0000 7f99edffb640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T11:20:56.178 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.201+0000 7f99edffb640 10 client.4583.objecter successfully canceled tick 2024-03-29T11:20:56.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.202+0000 7f99edffb640 1 -- 172.21.15.157:0/3305793498 >> v1:172.21.15.157:6816/1747058247 conn(0x7f99f406dd00 legacy=0x7f99f40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.202+0000 7f99edffb640 10 monclient: shutdown 2024-03-29T11:20:56.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.202+0000 7f99edffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:56.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.202+0000 7f99edffb640 1 -- 172.21.15.157:0/3305793498 >> v1:172.21.15.157:6790/0 conn(0x7f9a08142b10 legacy=0x7f9a08132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.202+0000 7f99edffb640 1 -- 172.21.15.157:0/3305793498 shutdown_connections 2024-03-29T11:20:56.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.202+0000 7f99edffb640 1 -- 172.21.15.157:0/3305793498 >> 172.21.15.157:0/3305793498 conn(0x7f9a08066a00 msgr2=0x7f9a08066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:56.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.203+0000 7f99edffb640 1 -- 172.21.15.157:0/3305793498 shutdown_connections 2024-03-29T11:20:56.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.203+0000 7f99edffb640 1 -- 172.21.15.157:0/3305793498 wait complete. 2024-03-29T11:20:56.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:20:56.203+0000 7f99edffb640 1 librados: shutdown 2024-03-29T11:20:56.199 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:03.007 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.181+0000 7f73ead0e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.181+0000 7f73ead0e640 10 monclient: build_initial_monmap 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.181+0000 7f73ead0e640 10 monclient: monmap: 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:21:03.182943+0000 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:21:03.182943+0000 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:03.159 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:21:03.160 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:21:03.160 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:03.160 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.182+0000 7f73ead0e640 1 Processor -- start 2024-03-29T11:21:03.160 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.183+0000 7f73ead0e640 1 -- start start 2024-03-29T11:21:03.160 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.183+0000 7f73ead0e640 10 monclient: init 2024-03-29T11:21:03.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.183+0000 7f73ead0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:03.161 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.183+0000 7f73ead0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.184+0000 7f73ead0e640 10 monclient(hunting): picked mon.noname-b con 0x7f73e40ae250 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.184+0000 7f73ead0e640 10 monclient(hunting): picked mon.noname-a con 0x7f73e413f970 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.184+0000 7f73ead0e640 10 monclient(hunting): picked mon.noname-c con 0x7f73e4142800 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.184+0000 7f73ead0e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e40afe00 con 0x7f73e413f970 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.184+0000 7f73ead0e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e4142c10 con 0x7f73e4142800 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.184+0000 7f73ead0e640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e40afb20 con 0x7f73e40ae250 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.184+0000 7f73ead0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.184+0000 7f73ead0e640 10 monclient(hunting): authenticate will time out at 1475.381551s 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.185+0000 7f73e3fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f73e413f970 0x7f73e41420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:43958/0 (socket says 172.21.15.157:43958) 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.185+0000 7f73e3fff640 1 -- 172.21.15.157:0/682353054 learned_addr learned my addr 172.21.15.157:0/682353054 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.185+0000 7f73e37fe640 1 -- 172.21.15.157:0/682353054 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263607669 0 0) 0x7f73e40afe00 con 0x7f73e413f970 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.185+0000 7f73e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.163 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.185+0000 7f73e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.164 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.185+0000 7f73e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.164 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.185+0000 7f73e37fe640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:21:03.164 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.185+0000 7f73e37fe640 1 -- 172.21.15.157:0/682353054 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73cc003520 con 0x7f73e413f970 2024-03-29T11:21:03.164 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.186+0000 7f73e37fe640 1 -- 172.21.15.157:0/682353054 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2896431144 0 0) 0x7f73cc003520 con 0x7f73e413f970 2024-03-29T11:21:03.164 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.186+0000 7f73e37fe640 1 -- 172.21.15.157:0/682353054 >> v1:172.21.15.157:6790/0 conn(0x7f73e40ae250 legacy=0x7f73e413f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:03.164 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.187+0000 7f73e37fe640 1 -- 172.21.15.157:0/682353054 >> v1:172.21.15.195:6789/0 conn(0x7f73e4142800 legacy=0x7f73e414da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 1 monclient: found mon.noname-a 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 1 -- 172.21.15.157:0/682353054 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e40aab80 con 0x7f73e413f970 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 1 -- 172.21.15.157:0/682353054 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73d4001a10 con 0x7f73e413f970 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73ead0e640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73ead0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:03.168 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 10 monclient: dump: 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73ead0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.189+0000 7f73e37fe640 1 -- 172.21.15.157:0/682353054 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73d4001ca0 con 0x7f73e413f970 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.190+0000 7f73e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.190+0000 7f73ead0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.190+0000 7f73ead0e640 4 set_mon_vals no callback set 2024-03-29T11:21:03.169 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.190+0000 7f73e37fe640 1 -- 172.21.15.157:0/682353054 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73d4002030 con 0x7f73e413f970 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.190+0000 7f73e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.190+0000 7f73e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.190+0000 7f73e37fe640 10 monclient: dump: 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:03.170 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.190+0000 7f73ead0e640 10 monclient: shutdown 2024-03-29T11:21:03.171 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.191+0000 7f73ead0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:03.171 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.191+0000 7f73ead0e640 1 -- 172.21.15.157:0/682353054 >> v1:172.21.15.157:6789/0 conn(0x7f73e413f970 legacy=0x7f73e41420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.171 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.191+0000 7f73ead0e640 1 -- 172.21.15.157:0/682353054 shutdown_connections 2024-03-29T11:21:03.171 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.191+0000 7f73ead0e640 1 -- 172.21.15.157:0/682353054 >> 172.21.15.157:0/682353054 conn(0x7f73e40b35e0 msgr2=0x7f73e40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:03.171 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.191+0000 7f73ead0e640 1 -- 172.21.15.157:0/682353054 shutdown_connections 2024-03-29T11:21:03.171 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.194+0000 7f73ead0e640 1 -- 172.21.15.157:0/682353054 wait complete. 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.195+0000 7f73ead0e640 10 monclient: build_initial_monmap 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.195+0000 7f73ead0e640 10 monclient: monmap: 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:21:03.173 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:03.174 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.196+0000 7f73ead0e640 1 librados: starting msgr at 2024-03-29T11:21:03.174 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.196+0000 7f73ead0e640 1 librados: starting objecter 2024-03-29T11:21:03.174 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.196+0000 7f73ead0e640 1 Processor -- start 2024-03-29T11:21:03.174 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.196+0000 7f73ead0e640 1 -- start start 2024-03-29T11:21:03.174 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.196+0000 7f73ead0e640 1 librados: setting wanted keys 2024-03-29T11:21:03.174 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.196+0000 7f73ead0e640 1 librados: calling monclient init 2024-03-29T11:21:03.174 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.196+0000 7f73ead0e640 10 monclient: init 2024-03-29T11:21:03.175 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.197+0000 7f73ead0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:03.175 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.197+0000 7f73ead0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:03.175 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.197+0000 7f73ead0e640 10 monclient(hunting): picked mon.noname-a con 0x7f73e40ae250 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:03.176 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.197+0000 7f73ead0e640 10 monclient(hunting): picked mon.noname-b con 0x7f73e413f970 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:03.176 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.197+0000 7f73ead0e640 10 monclient(hunting): picked mon.noname-c con 0x7f73e4142800 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:03.176 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.198+0000 7f73ead0e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e4141e80 con 0x7f73e40ae250 2024-03-29T11:21:03.176 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.198+0000 7f73ead0e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e41d76d0 con 0x7f73e413f970 2024-03-29T11:21:03.176 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.198+0000 7f73ead0e640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e41420d0 con 0x7f73e4142800 2024-03-29T11:21:03.176 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.198+0000 7f73ead0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:03.176 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.198+0000 7f73ead0e640 10 monclient(hunting): authenticate will time out at 1475.395691s 2024-03-29T11:21:03.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.200+0000 7f73e8aaa640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f73e40ae250 0x7f73e41d6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:43954/0 (socket says 172.21.15.157:43954) 2024-03-29T11:21:03.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.200+0000 7f73e8aaa640 1 -- 172.21.15.157:0/1470603201 learned_addr learned my addr 172.21.15.157:0/1470603201 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 client.?.objecter ms_handle_connect 0x7f73e40ae250 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 client.?.objecter ms_handle_connect 0x7f73e413f970 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 client.?.objecter ms_handle_connect 0x7f73e4142800 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833872308 0 0) 0x7f73e41420d0 con 0x7f73e4142800 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d0004eb0 con 0x7f73e4142800 2024-03-29T11:21:03.181 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285854205 0 0) 0x7f73e4141e80 con 0x7f73e40ae250 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.202+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e41420d0 con 0x7f73e40ae250 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.203+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101996686 0 0) 0x7f73e41d76d0 con 0x7f73e413f970 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.203+0000 7f73e17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.203+0000 7f73e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.203+0000 7f73e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.203+0000 7f73e17fa640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:21:03.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.203+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e4141e80 con 0x7f73e413f970 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3092914704 0 0) 0x7f73e41420d0 con 0x7f73e40ae250 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73e41d76d0 con 0x7f73e40ae250 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2558690011 0 0) 0x7f73e4141e80 con 0x7f73e413f970 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73d0006ae0 con 0x7f73e413f970 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73d40018d0 con 0x7f73e413f970 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 10 monclient(hunting): dump: 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:03.183 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73d8004000 con 0x7f73e40ae250 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.204+0000 7f73e17fa640 10 monclient(hunting): dump: 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:03.184 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2898575218 0 0) 0x7f73e41d76d0 con 0x7f73e40ae250 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 >> v1:172.21.15.157:6790/0 conn(0x7f73e4142800 legacy=0x7f73e41e2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 >> v1:172.21.15.195:6789/0 conn(0x7f73e413f970 legacy=0x7f73e4138cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 1 monclient: found mon.a 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:03.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e41daae0 con 0x7f73e40ae250 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 10 monclient: _finish_auth 0 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73ead0e640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73ead0e640 10 monclient: _renew_subs 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73ead0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73ead0e640 1 -- 172.21.15.157:0/1470603201 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e40ad0a0 con 0x7f73e40ae250 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73ead0e640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73ead0e640 10 monclient: _renew_subs 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73ead0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73ead0e640 1 -- 172.21.15.157:0/1470603201 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e41395a0 con 0x7f73e40ae250 2024-03-29T11:21:03.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.205+0000 7f73ead0e640 1 librados: init done 2024-03-29T11:21:03.188 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.208+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73d8000ed0 con 0x7f73e40ae250 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.208+0000 7f73e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.208+0000 7f73e2ffd640 4 set_mon_vals no callback set 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.208+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f73d8004000 con 0x7f73e40ae250 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.208+0000 7f73e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.208+0000 7f73e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.208+0000 7f73e17fa640 10 monclient: dump: 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:03.189 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:03.190 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:03.190 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:03.190 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.209+0000 7f73ead0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:03.190 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.209+0000 7f73ead0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:03.190 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.209+0000 7f73ead0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:03.190 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.209+0000 7f73ead0e640 1 -- 172.21.15.157:0/1470603201 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c4005080 con 0x7f73e40ae250 2024-03-29T11:21:03.190 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.210+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f73d8001180 con 0x7f73e40ae250 2024-03-29T11:21:03.190 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.211+0000 7f73e3fff640 10 monclient: get_auth_request con 0x7f73d006e3b0 auth_method 0 2024-03-29T11:21:03.190 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.211+0000 7f73e17fa640 10 client.4563.objecter ms_handle_connect 0x7f73d006e3b0 2024-03-29T11:21:03.190 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.212+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f73d8054550 con 0x7f73e40ae250 2024-03-29T11:21:03.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.212+0000 7f73e17fa640 10 client.4563.objecter ms_dispatch 0x7f73e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:03.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.212+0000 7f73e17fa640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:03.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.212+0000 7f73e17fa640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:03.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.212+0000 7f73e17fa640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T11:21:03.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.222+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73d80507f0 con 0x7f73e40ae250 2024-03-29T11:21:03.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.222+0000 7f73e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:03.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.222+0000 7f73e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:03.521 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.544+0000 7f73ead0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:03.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.544+0000 7f73ead0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:03.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.544+0000 7f73ead0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:03.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.545+0000 7f73ead0e640 1 -- 172.21.15.157:0/1470603201 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73c4005330 con 0x7f73e40ae250 2024-03-29T11:21:03.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.545+0000 7f73e17fa640 1 -- 172.21.15.157:0/1470603201 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f73d80507f0 con 0x7f73e40ae250 2024-03-29T11:21:03.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.545+0000 7f73e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:03.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.545+0000 7f73e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:03.523 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:21:03.523 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:03.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.549+0000 7f73ead0e640 10 librados: watch_flush enter 2024-03-29T11:21:03.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.549+0000 7f73ead0e640 10 librados: watch_flush exit 2024-03-29T11:21:03.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.549+0000 7f73ead0e640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T11:21:03.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.549+0000 7f73ead0e640 10 client.4563.objecter successfully canceled tick 2024-03-29T11:21:03.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.549+0000 7f73ead0e640 1 -- 172.21.15.157:0/1470603201 >> v1:172.21.15.157:6816/1747058247 conn(0x7f73d006e3b0 legacy=0x7f73d0070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.549+0000 7f73ead0e640 10 monclient: shutdown 2024-03-29T11:21:03.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.549+0000 7f73ead0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:03.526 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.549+0000 7f73ead0e640 1 -- 172.21.15.157:0/1470603201 >> v1:172.21.15.157:6789/0 conn(0x7f73e40ae250 legacy=0x7f73e41d6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.551+0000 7f73ead0e640 1 -- 172.21.15.157:0/1470603201 shutdown_connections 2024-03-29T11:21:03.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.551+0000 7f73ead0e640 1 -- 172.21.15.157:0/1470603201 >> 172.21.15.157:0/1470603201 conn(0x7f73e40b35e0 msgr2=0x7f73e40b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:03.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.552+0000 7f73ead0e640 1 -- 172.21.15.157:0/1470603201 shutdown_connections 2024-03-29T11:21:03.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.552+0000 7f73ead0e640 1 -- 172.21.15.157:0/1470603201 wait complete. 2024-03-29T11:21:03.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:03.552+0000 7f73ead0e640 1 librados: shutdown 2024-03-29T11:21:03.545 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:10.352 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.526+0000 7f4c46958640 10 monclient: get_monmap_and_config 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.526+0000 7f4c46958640 10 monclient: build_initial_monmap 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.526+0000 7f4c46958640 10 monclient: monmap: 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:21:10.527339+0000 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:21:10.527339+0000 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:10.503 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:21:10.504 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:21:10.504 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:10.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.527+0000 7f4c46958640 1 Processor -- start 2024-03-29T11:21:10.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.527+0000 7f4c46958640 1 -- start start 2024-03-29T11:21:10.504 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.527+0000 7f4c46958640 10 monclient: init 2024-03-29T11:21:10.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.528+0000 7f4c46958640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:10.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.528+0000 7f4c46958640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:10.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.528+0000 7f4c46958640 10 monclient(hunting): picked mon.noname-b con 0x7f4c380389e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:10.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.528+0000 7f4c46958640 10 monclient(hunting): picked mon.noname-a con 0x7f4c38038dc0 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:10.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.529+0000 7f4c46958640 10 monclient(hunting): picked mon.noname-c con 0x7f4c38030660 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:10.505 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.529+0000 7f4c46958640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c380c54c0 con 0x7f4c38038dc0 2024-03-29T11:21:10.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.529+0000 7f4c46958640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c38036430 con 0x7f4c38030660 2024-03-29T11:21:10.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.529+0000 7f4c46157640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4c38030660 0x7f4c380d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:55176/0 (socket says 172.21.15.157:55176) 2024-03-29T11:21:10.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.529+0000 7f4c46157640 1 -- 172.21.15.157:0/1281811324 learned_addr learned my addr 172.21.15.157:0/1281811324 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:10.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.529+0000 7f4c46958640 1 -- 172.21.15.157:0/1281811324 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c38039840 con 0x7f4c380389e0 2024-03-29T11:21:10.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.529+0000 7f4c46958640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:10.506 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.529+0000 7f4c46958640 10 monclient(hunting): authenticate will time out at 1482.726097s 2024-03-29T11:21:10.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.533+0000 7f4c44954640 1 -- 172.21.15.157:0/1281811324 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088706364 0 0) 0x7f4c38036430 con 0x7f4c38030660 2024-03-29T11:21:10.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.533+0000 7f4c44954640 10 monclient(hunting): _negotiate 2024-03-29T11:21:10.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.533+0000 7f4c44954640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:10.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.533+0000 7f4c44954640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:10.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.533+0000 7f4c44954640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:21:10.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.533+0000 7f4c44954640 1 -- 172.21.15.157:0/1281811324 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c2c003520 con 0x7f4c38030660 2024-03-29T11:21:10.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.533+0000 7f4c44954640 1 -- 172.21.15.157:0/1281811324 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3811843602 0 0) 0x7f4c2c003520 con 0x7f4c38030660 2024-03-29T11:21:10.510 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.533+0000 7f4c44954640 1 -- 172.21.15.157:0/1281811324 >> v1:172.21.15.157:6790/0 conn(0x7f4c380389e0 legacy=0x7f4c38033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:10.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 1 -- 172.21.15.157:0/1281811324 >> v1:172.21.15.157:6789/0 conn(0x7f4c38038dc0 legacy=0x7f4c380cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 10 monclient: _finish_hunting 0 2024-03-29T11:21:10.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 1 monclient: found mon.noname-c 2024-03-29T11:21:10.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:10.514 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 1 -- 172.21.15.157:0/1281811324 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38035690 con 0x7f4c38030660 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 10 monclient: _finish_auth 0 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 10 monclient: _check_auth_tickets 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 1 -- 172.21.15.157:0/1281811324 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4c30001690 con 0x7f4c38030660 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c44954640 10 monclient: dump: 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:10.515 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c46958640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.536+0000 7f4c46958640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.537+0000 7f4c44954640 1 -- 172.21.15.157:0/1281811324 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c30001920 con 0x7f4c38030660 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.537+0000 7f4c44954640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.537+0000 7f4c44954640 1 -- 172.21.15.157:0/1281811324 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4c30001cb0 con 0x7f4c38030660 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.537+0000 7f4c44954640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.537+0000 7f4c44954640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.537+0000 7f4c44954640 10 monclient: dump: 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:10.516 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:10.517 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:10.517 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.517 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:10.517 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:10.517 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:10.517 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:10.517 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.537+0000 7f4c46958640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.537+0000 7f4c46958640 4 set_mon_vals no callback set 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.538+0000 7f4c46958640 10 monclient: shutdown 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.538+0000 7f4c46958640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.538+0000 7f4c46958640 1 -- 172.21.15.157:0/1281811324 >> v1:172.21.15.195:6789/0 conn(0x7f4c38030660 legacy=0x7f4c380d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.538+0000 7f4c46958640 1 -- 172.21.15.157:0/1281811324 shutdown_connections 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.538+0000 7f4c46958640 1 -- 172.21.15.157:0/1281811324 >> 172.21.15.157:0/1281811324 conn(0x7f4c38005c00 msgr2=0x7f4c38006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.540+0000 7f4c46958640 1 -- 172.21.15.157:0/1281811324 shutdown_connections 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.540+0000 7f4c46958640 1 -- 172.21.15.157:0/1281811324 wait complete. 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.541+0000 7f4c46958640 10 monclient: build_initial_monmap 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.541+0000 7f4c46958640 10 monclient: monmap: 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:10.519 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.542+0000 7f4c46958640 1 librados: starting msgr at 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.542+0000 7f4c46958640 1 librados: starting objecter 2024-03-29T11:21:10.520 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.542+0000 7f4c46958640 1 Processor -- start 2024-03-29T11:21:10.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.542+0000 7f4c46958640 1 -- start start 2024-03-29T11:21:10.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.542+0000 7f4c46958640 1 librados: setting wanted keys 2024-03-29T11:21:10.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.542+0000 7f4c46958640 1 librados: calling monclient init 2024-03-29T11:21:10.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.542+0000 7f4c46958640 10 monclient: init 2024-03-29T11:21:10.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.544+0000 7f4c46958640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:10.522 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.544+0000 7f4c46958640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:10.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.544+0000 7f4c46958640 10 monclient(hunting): picked mon.noname-b con 0x7f4c38030660 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:10.523 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.544+0000 7f4c46958640 10 monclient(hunting): picked mon.noname-a con 0x7f4c380389e0 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:10.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.548+0000 7f4c46958640 10 monclient(hunting): picked mon.noname-c con 0x7f4c380c7670 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:10.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.548+0000 7f4c46958640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3802f9d0 con 0x7f4c380389e0 2024-03-29T11:21:10.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.548+0000 7f4c46958640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3802fb60 con 0x7f4c38030660 2024-03-29T11:21:10.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.548+0000 7f4c46958640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3802fcf0 con 0x7f4c380c7670 2024-03-29T11:21:10.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.548+0000 7f4c46958640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:10.525 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.548+0000 7f4c46958640 10 monclient(hunting): authenticate will time out at 1482.745529s 2024-03-29T11:21:10.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c46157640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f4c380c7670 0x7f4c3816c860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:54189/0 (socket says 172.21.15.157:54189) 2024-03-29T11:21:10.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c46157640 1 -- 172.21.15.157:0/3529266091 learned_addr learned my addr 172.21.15.157:0/3529266091 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:10.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 10 client.?.objecter ms_handle_connect 0x7f4c380c7670 2024-03-29T11:21:10.528 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941695116 0 0) 0x7f4c3802fcf0 con 0x7f4c380c7670 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c24004eb0 con 0x7f4c380c7670 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 10 client.?.objecter ms_handle_connect 0x7f4c38030660 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 10 client.?.objecter ms_handle_connect 0x7f4c380389e0 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.549+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4038458933 0 0) 0x7f4c24004eb0 con 0x7f4c380c7670 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c3802fcf0 con 0x7f4c380c7670 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4c30001550 con 0x7f4c380c7670 2024-03-29T11:21:10.529 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient(hunting): dump: 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:10.530 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282068834 0 0) 0x7f4c3802fb60 con 0x7f4c38030660 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c24004eb0 con 0x7f4c38030660 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2573262527 0 0) 0x7f4c3802fcf0 con 0x7f4c380c7670 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 >> v1:172.21.15.195:6789/0 conn(0x7f4c38030660 legacy=0x7f4c380c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 >> v1:172.21.15.157:6789/0 conn(0x7f4c380389e0 legacy=0x7f4c38161020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 1 monclient: found mon.c 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c380c3250 con 0x7f4c380c7670 2024-03-29T11:21:10.531 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c46958640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c46958640 10 monclient: _renew_subs 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c46958640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c46958640 1 -- 172.21.15.157:0/3529266091 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c380c34a0 con 0x7f4c380c7670 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c46958640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c46958640 10 monclient: _renew_subs 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c46958640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c46958640 1 -- 172.21.15.157:0/3529266091 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c38009990 con 0x7f4c380c7670 2024-03-29T11:21:10.532 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.550+0000 7f4c46958640 1 librados: init done 2024-03-29T11:21:10.533 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.552+0000 7f4c2bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:10.533 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c2bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c2bfff640 1 -- 172.21.15.157:0/3529266091 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c380c8620 con 0x7f4c380c7670 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c30004bf0 con 0x7f4c380c7670 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f4c30004f00 con 0x7f4c380c7670 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c367fc640 10 monclient: dump: 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:10.534 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f4c3001aed0 con 0x7f4c380c7670 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.553+0000 7f4c377fe640 4 set_mon_vals no callback set 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.554+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f4c30084bf0 con 0x7f4c380c7670 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.554+0000 7f4c367fc640 10 client.4589.objecter ms_dispatch 0x7f4c40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.554+0000 7f4c367fc640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.554+0000 7f4c367fc640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.554+0000 7f4c367fc640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.555+0000 7f4c45956640 10 monclient: get_auth_request con 0x7f4c2406de00 auth_method 0 2024-03-29T11:21:10.535 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.555+0000 7f4c367fc640 10 client.4589.objecter ms_handle_connect 0x7f4c2406de00 2024-03-29T11:21:10.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.562+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c30050550 con 0x7f4c380c7670 2024-03-29T11:21:10.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.562+0000 7f4c367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:10.538 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.562+0000 7f4c367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:10.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.892+0000 7f4c2bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:10.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.892+0000 7f4c2bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:10.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.892+0000 7f4c2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:21:10.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.892+0000 7f4c2bfff640 1 -- 172.21.15.157:0/3529266091 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c38007b90 con 0x7f4c380c7670 2024-03-29T11:21:10.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.893+0000 7f4c367fc640 1 -- 172.21.15.157:0/3529266091 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f4c30052480 con 0x7f4c380c7670 2024-03-29T11:21:10.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.893+0000 7f4c367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:10.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.893+0000 7f4c367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:10.870 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:21:10.871 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:10.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.897+0000 7f4c2bfff640 10 librados: watch_flush enter 2024-03-29T11:21:10.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.897+0000 7f4c2bfff640 10 librados: watch_flush exit 2024-03-29T11:21:10.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.897+0000 7f4c2bfff640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:21:10.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.897+0000 7f4c2bfff640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:21:10.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.897+0000 7f4c2bfff640 1 -- 172.21.15.157:0/3529266091 >> v1:172.21.15.157:6816/1747058247 conn(0x7f4c2406de00 legacy=0x7f4c240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.897+0000 7f4c2bfff640 10 monclient: shutdown 2024-03-29T11:21:10.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.897+0000 7f4c2bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:10.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.897+0000 7f4c2bfff640 1 -- 172.21.15.157:0/3529266091 >> v1:172.21.15.157:6790/0 conn(0x7f4c380c7670 legacy=0x7f4c3816c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.874 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.897+0000 7f4c2bfff640 1 -- 172.21.15.157:0/3529266091 shutdown_connections 2024-03-29T11:21:10.875 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.897+0000 7f4c2bfff640 1 -- 172.21.15.157:0/3529266091 >> 172.21.15.157:0/3529266091 conn(0x7f4c38005c00 msgr2=0x7f4c38034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:10.875 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.899+0000 7f4c2bfff640 1 -- 172.21.15.157:0/3529266091 shutdown_connections 2024-03-29T11:21:10.875 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.899+0000 7f4c2bfff640 1 -- 172.21.15.157:0/3529266091 wait complete. 2024-03-29T11:21:10.877 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:10.901+0000 7f4c2bfff640 1 librados: shutdown 2024-03-29T11:21:10.895 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:17.702 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:17.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.871+0000 7f19bc50e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:17.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.871+0000 7f19bc50e640 10 monclient: build_initial_monmap 2024-03-29T11:21:17.848 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.871+0000 7f19bc50e640 10 monclient: monmap: 2024-03-29T11:21:17.848 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:17.848 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:17.848 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:21:17.872672+0000 2024-03-29T11:21:17.848 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:21:17.872672+0000 2024-03-29T11:21:17.848 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:17.849 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:17.849 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:17.849 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:21:17.849 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:21:17.849 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:17.849 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.872+0000 7f19bc50e640 1 Processor -- start 2024-03-29T11:21:17.849 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.872+0000 7f19bc50e640 1 -- start start 2024-03-29T11:21:17.849 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.873+0000 7f19bc50e640 10 monclient: init 2024-03-29T11:21:17.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.874+0000 7f19bc50e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:17.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.874+0000 7f19bc50e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:17.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.874+0000 7f19bc50e640 10 monclient(hunting): picked mon.noname-c con 0x7f19b41422b0 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:17.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.874+0000 7f19ba2aa640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f19b41422b0 0x7f19b413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:42118/0 (socket says 172.21.15.157:42118) 2024-03-29T11:21:17.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.874+0000 7f19ba2aa640 1 -- 172.21.15.157:0/1378604016 learned_addr learned my addr 172.21.15.157:0/1378604016 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:17.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.874+0000 7f19bc50e640 10 monclient(hunting): picked mon.noname-b con 0x7f19b4142690 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:17.851 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.874+0000 7f19bc50e640 10 monclient(hunting): picked mon.noname-a con 0x7f19b40b17d0 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:17.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19bc50e640 1 -- 172.21.15.157:0/1378604016 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b4142c80 con 0x7f19b40b17d0 2024-03-29T11:21:17.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19bc50e640 1 -- 172.21.15.157:0/1378604016 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b40aae40 con 0x7f19b41422b0 2024-03-29T11:21:17.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19bc50e640 1 -- 172.21.15.157:0/1378604016 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b413fbd0 con 0x7f19b4142690 2024-03-29T11:21:17.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19bc50e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:17.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19bc50e640 10 monclient(hunting): authenticate will time out at 1490.071945s 2024-03-29T11:21:17.854 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263373654 0 0) 0x7f19b40aae40 con 0x7f19b41422b0 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 10 monclient(hunting): _negotiate 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a0003520 con 0x7f19b41422b0 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667614208 0 0) 0x7f19b4142c80 con 0x7f19b40b17d0 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 10 monclient(hunting): _negotiate 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b40aae40 con 0x7f19b40b17d0 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1443595154 0 0) 0x7f19a0003520 con 0x7f19b41422b0 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 >> v1:172.21.15.157:6790/0 conn(0x7f19b4142690 legacy=0x7f19b40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.855 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 >> v1:172.21.15.157:6789/0 conn(0x7f19b40b17d0 legacy=0x7f19b40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 10 monclient: _finish_hunting 0 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 1 monclient: found mon.noname-c 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.875+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b4151940 con 0x7f19b41422b0 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.876+0000 7f19b92a8640 10 monclient: _finish_auth 0 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.876+0000 7f19b92a8640 10 monclient: _check_auth_tickets 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.876+0000 7f19b92a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.876+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f19ac001610 con 0x7f19b41422b0 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.876+0000 7f19b92a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.876+0000 7f19b92a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.876+0000 7f19b92a8640 10 monclient: dump: 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:17.856 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.876+0000 7f19bc50e640 5 monclient: authenticate success, global_id 4564 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.876+0000 7f19bc50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.877+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19ac001830 con 0x7f19b41422b0 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.877+0000 7f19b92a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.877+0000 7f19bc50e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.877+0000 7f19bc50e640 4 set_mon_vals no callback set 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.877+0000 7f19b92a8640 1 -- 172.21.15.157:0/1378604016 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f19ac001bc0 con 0x7f19b41422b0 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.877+0000 7f19b92a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.857 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.877+0000 7f19b92a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.877+0000 7f19b92a8640 10 monclient: dump: 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:17.858 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.878+0000 7f19bc50e640 10 monclient: shutdown 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.878+0000 7f19bc50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.878+0000 7f19bc50e640 1 -- 172.21.15.157:0/1378604016 >> v1:172.21.15.195:6789/0 conn(0x7f19b41422b0 legacy=0x7f19b413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.878+0000 7f19bc50e640 1 -- 172.21.15.157:0/1378604016 shutdown_connections 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.878+0000 7f19bc50e640 1 -- 172.21.15.157:0/1378604016 >> 172.21.15.157:0/1378604016 conn(0x7f19b4066a00 msgr2=0x7f19b4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.878+0000 7f19bc50e640 1 -- 172.21.15.157:0/1378604016 shutdown_connections 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.878+0000 7f19bc50e640 1 -- 172.21.15.157:0/1378604016 wait complete. 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.879+0000 7f19bc50e640 10 monclient: build_initial_monmap 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.879+0000 7f19bc50e640 10 monclient: monmap: 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:21:17.864 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.880+0000 7f19bc50e640 1 librados: starting msgr at 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.880+0000 7f19bc50e640 1 librados: starting objecter 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.880+0000 7f19bc50e640 1 Processor -- start 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.880+0000 7f19bc50e640 1 -- start start 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.880+0000 7f19bc50e640 1 librados: setting wanted keys 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.880+0000 7f19bc50e640 1 librados: calling monclient init 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.880+0000 7f19bc50e640 10 monclient: init 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19bc50e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:17.865 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19bc50e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19bc50e640 10 monclient(hunting): picked mon.noname-c con 0x7f19b40b17d0 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19bc50e640 10 monclient(hunting): picked mon.noname-b con 0x7f19b41422b0 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19bc50e640 10 monclient(hunting): picked mon.noname-a con 0x7f19b4142690 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19bc50e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b40b73f0 con 0x7f19b4142690 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19bc50e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b40b67a0 con 0x7f19b41422b0 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19bc50e640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b41509b0 con 0x7f19b40b17d0 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19bc50e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19bc50e640 10 monclient(hunting): authenticate will time out at 1490.078175s 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19ba2aa640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f19b40b17d0 0x7f19b414f280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:48014/0 (socket says 172.21.15.157:48014) 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19ba2aa640 1 -- 172.21.15.157:0/906733692 learned_addr learned my addr 172.21.15.157:0/906733692 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19aaffd640 10 client.?.objecter ms_handle_connect 0x7f19b4142690 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19aaffd640 10 client.?.objecter ms_handle_connect 0x7f19b40b17d0 2024-03-29T11:21:17.866 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.881+0000 7f19aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 client.?.objecter ms_handle_connect 0x7f19b41422b0 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926108111 0 0) 0x7f19b41509b0 con 0x7f19b40b17d0 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1998004eb0 con 0x7f19b40b17d0 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402402300 0 0) 0x7f19b40b73f0 con 0x7f19b4142690 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b41509b0 con 0x7f19b4142690 2024-03-29T11:21:17.867 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2431354516 0 0) 0x7f1998004eb0 con 0x7f19b40b17d0 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19b40b73f0 con 0x7f19b40b17d0 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f19ac001550 con 0x7f19b40b17d0 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): dump: 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:17.868 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3177250154 0 0) 0x7f19b40b73f0 con 0x7f19b40b17d0 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 >> v1:172.21.15.195:6789/0 conn(0x7f19b41422b0 legacy=0x7f19b412d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 >> v1:172.21.15.157:6789/0 conn(0x7f19b4142690 legacy=0x7f19b412dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 monclient: found mon.c 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.882+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b412e700 con 0x7f19b40b17d0 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19bc50e640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:21:17.869 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19bc50e640 10 monclient: _renew_subs 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19bc50e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19bc50e640 1 -- 172.21.15.157:0/906733692 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b40b6a10 con 0x7f19b40b17d0 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19bc50e640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19bc50e640 10 monclient: _renew_subs 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19bc50e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19bc50e640 1 -- 172.21.15.157:0/906733692 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b41506d0 con 0x7f19b40b17d0 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19bc50e640 1 librados: init done 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19ac004a90 con 0x7f19b40b17d0 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f19ac004da0 con 0x7f19b40b17d0 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19aaffd640 10 monclient: dump: 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:17.870 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.883+0000 7f19abfff640 4 set_mon_vals no callback set 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.885+0000 7f19bc50e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.885+0000 7f19bc50e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.885+0000 7f19bc50e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.885+0000 7f19bc50e640 1 -- 172.21.15.157:0/906733692 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199c005080 con 0x7f19b40b17d0 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.885+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f19ac01ad70 con 0x7f19b40b17d0 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.886+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f19ac084b90 con 0x7f19b40b17d0 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.886+0000 7f19aaffd640 10 client.4595.objecter ms_dispatch 0x7f19b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:17.871 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.886+0000 7f19aaffd640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:17.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.886+0000 7f19aaffd640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:17.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.886+0000 7f19aaffd640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T11:21:17.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.886+0000 7f19b9aa9640 10 monclient: get_auth_request con 0x7f199806dce0 auth_method 0 2024-03-29T11:21:17.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.886+0000 7f19aaffd640 10 client.4595.objecter ms_handle_connect 0x7f199806dce0 2024-03-29T11:21:17.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.895+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19ac050470 con 0x7f19b40b17d0 2024-03-29T11:21:17.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.895+0000 7f19aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:17.872 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:17.895+0000 7f19aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:18.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.220+0000 7f19bc50e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:18.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.220+0000 7f19bc50e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:18.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.220+0000 7f19bc50e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:21:18.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.221+0000 7f19bc50e640 1 -- 172.21.15.157:0/906733692 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f199c005330 con 0x7f19b40b17d0 2024-03-29T11:21:18.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.222+0000 7f19aaffd640 1 -- 172.21.15.157:0/906733692 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f19ac0523a0 con 0x7f19b40b17d0 2024-03-29T11:21:18.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.222+0000 7f19aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:18.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.222+0000 7f19aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:18.199 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:21:18.199 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:18.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.226+0000 7f19bc50e640 10 librados: watch_flush enter 2024-03-29T11:21:18.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 10 librados: watch_flush exit 2024-03-29T11:21:18.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T11:21:18.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 10 client.4595.objecter successfully canceled tick 2024-03-29T11:21:18.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 1 -- 172.21.15.157:0/906733692 >> v1:172.21.15.157:6816/1747058247 conn(0x7f199806dce0 legacy=0x7f19980701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:18.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 10 monclient: shutdown 2024-03-29T11:21:18.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:18.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 1 -- 172.21.15.157:0/906733692 >> v1:172.21.15.157:6790/0 conn(0x7f19b40b17d0 legacy=0x7f19b414f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:18.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 1 -- 172.21.15.157:0/906733692 shutdown_connections 2024-03-29T11:21:18.204 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 1 -- 172.21.15.157:0/906733692 >> 172.21.15.157:0/906733692 conn(0x7f19b4066a00 msgr2=0x7f19b4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:18.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 1 -- 172.21.15.157:0/906733692 shutdown_connections 2024-03-29T11:21:18.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 1 -- 172.21.15.157:0/906733692 wait complete. 2024-03-29T11:21:18.205 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:18.227+0000 7f19bc50e640 1 librados: shutdown 2024-03-29T11:21:18.218 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:25.025 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:25.177 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.200+0000 7fbd454ae640 10 monclient: get_monmap_and_config 2024-03-29T11:21:25.177 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.200+0000 7fbd454ae640 10 monclient: build_initial_monmap 2024-03-29T11:21:25.177 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.200+0000 7fbd454ae640 10 monclient: monmap: 2024-03-29T11:21:25.178 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:25.178 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:25.178 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:21:25.201972+0000 2024-03-29T11:21:25.178 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:21:25.201972+0000 2024-03-29T11:21:25.178 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:25.178 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:25.178 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:25.178 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:21:25.178 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:21:25.178 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:25.179 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.201+0000 7fbd454ae640 1 Processor -- start 2024-03-29T11:21:25.179 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.201+0000 7fbd454ae640 1 -- start start 2024-03-29T11:21:25.179 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.201+0000 7fbd454ae640 10 monclient: init 2024-03-29T11:21:25.179 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.202+0000 7fbd454ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:25.179 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.202+0000 7fbd454ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:25.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.202+0000 7fbd454ae640 10 monclient(hunting): picked mon.noname-b con 0x7fbd401457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:25.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.202+0000 7fbd454ae640 10 monclient(hunting): picked mon.noname-c con 0x7fbd4013f650 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:25.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.202+0000 7fbd454ae640 10 monclient(hunting): picked mon.noname-a con 0x7fbd400a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:25.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.202+0000 7fbd454ae640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd400aaef0 con 0x7fbd400a9d70 2024-03-29T11:21:25.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd454ae640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd40145bd0 con 0x7fbd4013f650 2024-03-29T11:21:25.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd454ae640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd40145eb0 con 0x7fbd401457f0 2024-03-29T11:21:25.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd454ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:25.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd454ae640 10 monclient(hunting): authenticate will time out at 1497.399890s 2024-03-29T11:21:25.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3e7fc640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fbd4013f650 0x7fbd400b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:39762/0 (socket says 172.21.15.157:39762) 2024-03-29T11:21:25.180 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3e7fc640 1 -- 172.21.15.157:0/4134669102 learned_addr learned my addr 172.21.15.157:0/4134669102 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:25.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815673078 0 0) 0x7fbd40145eb0 con 0x7fbd401457f0 2024-03-29T11:21:25.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.182 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd28003520 con 0x7fbd401457f0 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781690205 0 0) 0x7fbd400aaef0 con 0x7fbd400a9d70 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.203+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd40145eb0 con 0x7fbd400a9d70 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2134933693 0 0) 0x7fbd28003520 con 0x7fbd401457f0 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 >> v1:172.21.15.195:6789/0 conn(0x7fbd4013f650 legacy=0x7fbd400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 >> v1:172.21.15.157:6789/0 conn(0x7fbd400a9d70 legacy=0x7fbd4014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 1 monclient: found mon.noname-b 2024-03-29T11:21:25.183 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd40146340 con 0x7fbd401457f0 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd454ae640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd454ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fbd300015c0 con 0x7fbd401457f0 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: dump: 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:25.184 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd454ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd30001720 con 0x7fbd401457f0 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd454ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd454ae640 4 set_mon_vals no callback set 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 1 -- 172.21.15.157:0/4134669102 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fbd30001b30 con 0x7fbd401457f0 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.185 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.204+0000 7fbd3dffb640 10 monclient: dump: 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.205+0000 7fbd454ae640 10 monclient: shutdown 2024-03-29T11:21:25.186 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.205+0000 7fbd454ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:25.187 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.205+0000 7fbd454ae640 1 -- 172.21.15.157:0/4134669102 >> v1:172.21.15.157:6790/0 conn(0x7fbd401457f0 legacy=0x7fbd4013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.187 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.205+0000 7fbd454ae640 1 -- 172.21.15.157:0/4134669102 shutdown_connections 2024-03-29T11:21:25.187 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.205+0000 7fbd454ae640 1 -- 172.21.15.157:0/4134669102 >> 172.21.15.157:0/4134669102 conn(0x7fbd40066a00 msgr2=0x7fbd40066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:25.187 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.205+0000 7fbd454ae640 1 -- 172.21.15.157:0/4134669102 shutdown_connections 2024-03-29T11:21:25.187 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.205+0000 7fbd454ae640 1 -- 172.21.15.157:0/4134669102 wait complete. 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.206+0000 7fbd454ae640 10 monclient: build_initial_monmap 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.206+0000 7fbd454ae640 10 monclient: monmap: 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:21:25.190 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.207+0000 7fbd454ae640 1 librados: starting msgr at 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.207+0000 7fbd454ae640 1 librados: starting objecter 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.207+0000 7fbd454ae640 1 Processor -- start 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.207+0000 7fbd454ae640 1 -- start start 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.207+0000 7fbd454ae640 1 librados: setting wanted keys 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.207+0000 7fbd454ae640 1 librados: calling monclient init 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.207+0000 7fbd454ae640 10 monclient: init 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd454ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd454ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd454ae640 10 monclient(hunting): picked mon.noname-b con 0x7fbd400a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd454ae640 10 monclient(hunting): picked mon.noname-a con 0x7fbd4013f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd454ae640 10 monclient(hunting): picked mon.noname-c con 0x7fbd401457f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd454ae640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4013b7a0 con 0x7fbd4013f650 2024-03-29T11:21:25.191 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd454ae640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd40141de0 con 0x7fbd400a9d70 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd454ae640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd401420c0 con 0x7fbd401457f0 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd454ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd454ae640 10 monclient(hunting): authenticate will time out at 1497.405589s 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd3e7fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fbd4013f650 0x7fbd40135060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:58654/0 (socket says 172.21.15.157:58654) 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.208+0000 7fbd3e7fc640 1 -- 172.21.15.157:0/1545635766 learned_addr learned my addr 172.21.15.157:0/1545635766 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 client.?.objecter ms_handle_connect 0x7fbd401457f0 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 client.?.objecter ms_handle_connect 0x7fbd4013f650 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 client.?.objecter ms_handle_connect 0x7fbd400a9d70 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335763509 0 0) 0x7fbd4013b7a0 con 0x7fbd4013f650 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.192 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd2c004eb0 con 0x7fbd4013f650 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503814376 0 0) 0x7fbd401420c0 con 0x7fbd401457f0 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd4013b7a0 con 0x7fbd401457f0 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026537640 0 0) 0x7fbd40141de0 con 0x7fbd400a9d70 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:21:25.193 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.209+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd401420c0 con 0x7fbd400a9d70 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 226728163 0 0) 0x7fbd2c004eb0 con 0x7fbd4013f650 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd40141de0 con 0x7fbd4013f650 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fbd34001510 con 0x7fbd4013f650 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient(hunting): dump: 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:25.194 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3515207256 0 0) 0x7fbd4013b7a0 con 0x7fbd401457f0 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd2c004eb0 con 0x7fbd401457f0 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fbd38001c70 con 0x7fbd401457f0 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient(hunting): dump: 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:25.195 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1625181564 0 0) 0x7fbd40141de0 con 0x7fbd4013f650 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 >> v1:172.21.15.157:6790/0 conn(0x7fbd401457f0 legacy=0x7fbd401391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 >> v1:172.21.15.195:6789/0 conn(0x7fbd400a9d70 legacy=0x7fbd40142850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 monclient: found mon.a 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4013b9f0 con 0x7fbd4013f650 2024-03-29T11:21:25.196 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd454ae640 5 monclient: authenticate success, global_id 4581 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd454ae640 10 monclient: _renew_subs 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd454ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd454ae640 1 -- 172.21.15.157:0/1545635766 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd40140ec0 con 0x7fbd4013f650 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd454ae640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd454ae640 10 monclient: _renew_subs 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd454ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd454ae640 1 -- 172.21.15.157:0/1545635766 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd40135950 con 0x7fbd4013f650 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.210+0000 7fbd454ae640 1 librados: init done 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.212+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd34001a20 con 0x7fbd4013f650 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.212+0000 7fbd277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:25.197 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.212+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fbd34004ba0 con 0x7fbd4013f650 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.212+0000 7fbd277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.212+0000 7fbd3d7fa640 4 set_mon_vals no callback set 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.212+0000 7fbd277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.212+0000 7fbd277fe640 10 monclient: dump: 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:25.198 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:25.199 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:25.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.212+0000 7fbd454ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:25.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.213+0000 7fbd454ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:25.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.213+0000 7fbd454ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:25.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.213+0000 7fbd454ae640 1 -- 172.21.15.157:0/1545635766 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd20005080 con 0x7fbd4013f650 2024-03-29T11:21:25.199 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.214+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fbd3401ab70 con 0x7fbd4013f650 2024-03-29T11:21:25.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.214+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7fbd340847e0 con 0x7fbd4013f650 2024-03-29T11:21:25.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.214+0000 7fbd277fe640 10 client.4581.objecter ms_dispatch 0x7fbd40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:25.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.214+0000 7fbd277fe640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:25.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.214+0000 7fbd277fe640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:25.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.214+0000 7fbd277fe640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-29T11:21:25.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.214+0000 7fbd3effd640 10 monclient: get_auth_request con 0x7fbd2c06e0e0 auth_method 0 2024-03-29T11:21:25.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.214+0000 7fbd277fe640 10 client.4581.objecter ms_handle_connect 0x7fbd2c06e0e0 2024-03-29T11:21:25.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.222+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd34050030 con 0x7fbd4013f650 2024-03-29T11:21:25.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.222+0000 7fbd277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:25.200 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.222+0000 7fbd277fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:25.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.596+0000 7fbd454ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:25.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.596+0000 7fbd454ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:25.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.596+0000 7fbd454ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:25.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.596+0000 7fbd454ae640 1 -- 172.21.15.157:0/1545635766 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd20005330 con 0x7fbd4013f650 2024-03-29T11:21:25.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.597+0000 7fbd277fe640 1 -- 172.21.15.157:0/1545635766 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7fbd34052130 con 0x7fbd4013f650 2024-03-29T11:21:25.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.597+0000 7fbd277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:25.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.597+0000 7fbd277fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:25.574 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:21:25.574 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:25.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 10 librados: watch_flush enter 2024-03-29T11:21:25.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 10 librados: watch_flush exit 2024-03-29T11:21:25.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-29T11:21:25.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 10 client.4581.objecter successfully canceled tick 2024-03-29T11:21:25.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 1 -- 172.21.15.157:0/1545635766 >> v1:172.21.15.157:6816/1747058247 conn(0x7fbd2c06e0e0 legacy=0x7fbd2c0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 10 monclient: shutdown 2024-03-29T11:21:25.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:25.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 1 -- 172.21.15.157:0/1545635766 >> v1:172.21.15.157:6789/0 conn(0x7fbd4013f650 legacy=0x7fbd40135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 1 -- 172.21.15.157:0/1545635766 shutdown_connections 2024-03-29T11:21:25.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 1 -- 172.21.15.157:0/1545635766 >> 172.21.15.157:0/1545635766 conn(0x7fbd40066a00 msgr2=0x7fbd400b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:25.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.601+0000 7fbd454ae640 1 -- 172.21.15.157:0/1545635766 shutdown_connections 2024-03-29T11:21:25.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.602+0000 7fbd454ae640 1 -- 172.21.15.157:0/1545635766 wait complete. 2024-03-29T11:21:25.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:25.602+0000 7fbd454ae640 1 librados: shutdown 2024-03-29T11:21:25.593 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:32.401 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:32.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.581+0000 7f6af348f640 10 monclient: get_monmap_and_config 2024-03-29T11:21:32.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.581+0000 7f6af348f640 10 monclient: build_initial_monmap 2024-03-29T11:21:32.558 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.581+0000 7f6af348f640 10 monclient: monmap: 2024-03-29T11:21:32.558 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:32.558 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:32.558 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:21:32.582305+0000 2024-03-29T11:21:32.558 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:21:32.582305+0000 2024-03-29T11:21:32.558 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:32.559 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:32.559 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:32.559 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:21:32.559 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:21:32.559 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:32.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.582+0000 7f6af348f640 1 Processor -- start 2024-03-29T11:21:32.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.582+0000 7f6af348f640 1 -- start start 2024-03-29T11:21:32.559 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.582+0000 7f6af348f640 10 monclient: init 2024-03-29T11:21:32.563 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.583+0000 7f6af348f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:32.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.583+0000 7f6af348f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:32.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.583+0000 7f6af348f640 10 monclient(hunting): picked mon.noname-c con 0x7f6aec0b4330 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:32.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.583+0000 7f6af348f640 10 monclient(hunting): picked mon.noname-a con 0x7f6aec1470b0 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:32.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.583+0000 7f6af348f640 10 monclient(hunting): picked mon.noname-b con 0x7f6aec1439b0 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:32.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.583+0000 7f6af348f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec06f2a0 con 0x7f6aec1470b0 2024-03-29T11:21:32.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.583+0000 7f6af348f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec146b90 con 0x7f6aec0b4330 2024-03-29T11:21:32.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.583+0000 7f6af348f640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec0b2ca0 con 0x7f6aec1439b0 2024-03-29T11:21:32.565 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.583+0000 7f6af348f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.583+0000 7f6af348f640 10 monclient(hunting): authenticate will time out at 1504.780089s 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.587+0000 7f6af122b640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6aec0b4330 0x7f6aec0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:59407/0 (socket says 172.21.15.157:59407) 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.587+0000 7f6af122b640 1 -- 172.21.15.157:0/3670988277 learned_addr learned my addr 172.21.15.157:0/3670988277 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.587+0000 7f6aebfff640 1 -- 172.21.15.157:0/3670988277 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186094248 0 0) 0x7f6aec146b90 con 0x7f6aec0b4330 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.587+0000 7f6aebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.587+0000 7f6aebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.587+0000 7f6aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.587+0000 7f6aebfff640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.587+0000 7f6aebfff640 1 -- 172.21.15.157:0/3670988277 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad4003520 con 0x7f6aec0b4330 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.588+0000 7f6aebfff640 1 -- 172.21.15.157:0/3670988277 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2426184723 0 0) 0x7f6ad4003520 con 0x7f6aec0b4330 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.588+0000 7f6aebfff640 1 -- 172.21.15.157:0/3670988277 >> v1:172.21.15.157:6790/0 conn(0x7f6aec1439b0 legacy=0x7f6aec151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:32.566 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.588+0000 7f6aebfff640 1 -- 172.21.15.157:0/3670988277 >> v1:172.21.15.157:6789/0 conn(0x7f6aec1470b0 legacy=0x7f6aec0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:32.567 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:32.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 1 monclient: found mon.noname-c 2024-03-29T11:21:32.570 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 1 -- 172.21.15.157:0/3670988277 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec06e0c0 con 0x7f6aec0b4330 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 10 monclient: _finish_auth 0 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6af348f640 5 monclient: authenticate success, global_id 4573 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6af348f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 1 -- 172.21.15.157:0/3670988277 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6adc001690 con 0x7f6aec0b4330 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:32.571 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 10 monclient: dump: 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6af348f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 1 -- 172.21.15.157:0/3670988277 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6adc001920 con 0x7f6aec0b4330 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6aebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6af348f640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:32.572 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.591+0000 7f6af348f640 4 set_mon_vals no callback set 2024-03-29T11:21:32.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.592+0000 7f6af348f640 10 monclient: shutdown 2024-03-29T11:21:32.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.592+0000 7f6af348f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:32.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.592+0000 7f6af348f640 1 -- 172.21.15.157:0/3670988277 >> v1:172.21.15.195:6789/0 conn(0x7f6aec0b4330 legacy=0x7f6aec0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.592+0000 7f6af348f640 1 -- 172.21.15.157:0/3670988277 shutdown_connections 2024-03-29T11:21:32.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.592+0000 7f6af348f640 1 -- 172.21.15.157:0/3670988277 >> 172.21.15.157:0/3670988277 conn(0x7f6aec066a00 msgr2=0x7f6aec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:32.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.596+0000 7f6af348f640 1 -- 172.21.15.157:0/3670988277 shutdown_connections 2024-03-29T11:21:32.573 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.596+0000 7f6af348f640 1 -- 172.21.15.157:0/3670988277 wait complete. 2024-03-29T11:21:32.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.597+0000 7f6af348f640 10 monclient: build_initial_monmap 2024-03-29T11:21:32.574 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.597+0000 7f6af348f640 10 monclient: monmap: 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.598+0000 7f6af348f640 1 librados: starting msgr at 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.598+0000 7f6af348f640 1 librados: starting objecter 2024-03-29T11:21:32.575 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.598+0000 7f6af348f640 1 Processor -- start 2024-03-29T11:21:32.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 1 -- start start 2024-03-29T11:21:32.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 1 librados: setting wanted keys 2024-03-29T11:21:32.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 1 librados: calling monclient init 2024-03-29T11:21:32.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 10 monclient: init 2024-03-29T11:21:32.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:32.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:32.578 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 10 monclient(hunting): picked mon.noname-a con 0x7f6aec0b4330 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:32.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 10 monclient(hunting): picked mon.noname-c con 0x7f6aec1439b0 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:32.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 10 monclient(hunting): picked mon.noname-b con 0x7f6aec142cc0 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:32.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec1df250 con 0x7f6aec0b4330 2024-03-29T11:21:32.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec143760 con 0x7f6aec142cc0 2024-03-29T11:21:32.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec1e8e40 con 0x7f6aec1439b0 2024-03-29T11:21:32.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:32.579 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.599+0000 7f6af348f640 10 monclient(hunting): authenticate will time out at 1504.796807s 2024-03-29T11:21:32.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.603+0000 7f6af1a2c640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6aec142cc0 0x7f6aec1205b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:59517/0 (socket says 172.21.15.157:59517) 2024-03-29T11:21:32.580 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.604+0000 7f6af1a2c640 1 -- 172.21.15.157:0/1109265884 learned_addr learned my addr 172.21.15.157:0/1109265884 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:32.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.605+0000 7f6ae9ffb640 10 client.?.objecter ms_handle_connect 0x7f6aec0b4330 2024-03-29T11:21:32.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.605+0000 7f6ae9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:32.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.605+0000 7f6ae9ffb640 10 client.?.objecter ms_handle_connect 0x7f6aec142cc0 2024-03-29T11:21:32.582 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.605+0000 7f6ae9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:32.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.606+0000 7f6ae9ffb640 10 client.?.objecter ms_handle_connect 0x7f6aec1439b0 2024-03-29T11:21:32.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.606+0000 7f6ae9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:32.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.606+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723621836 0 0) 0x7f6aec1df250 con 0x7f6aec0b4330 2024-03-29T11:21:32.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.606+0000 7f6ae9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.585 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.606+0000 7f6ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.606+0000 7f6ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.606+0000 7f6ae9ffb640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.606+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad8004eb0 con 0x7f6aec0b4330 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.606+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441892032 0 0) 0x7f6aec143760 con 0x7f6aec142cc0 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aec1df250 con 0x7f6aec142cc0 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 709020138 0 0) 0x7f6ad8004eb0 con 0x7f6aec0b4330 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6aec143760 con 0x7f6aec0b4330 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6adc004980 con 0x7f6aec0b4330 2024-03-29T11:21:32.586 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:32.587 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765040466 0 0) 0x7f6aec1e8e40 con 0x7f6aec1439b0 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.607+0000 7f6ae9ffb640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad8004eb0 con 0x7f6aec1439b0 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4208755476 0 0) 0x7f6aec143760 con 0x7f6aec0b4330 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 >> v1:172.21.15.157:6790/0 conn(0x7f6aec1439b0 legacy=0x7f6aec11fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 >> v1:172.21.15.195:6789/0 conn(0x7f6aec142cc0 legacy=0x7f6aec1205b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 1 monclient: found mon.a 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec1e9fe0 con 0x7f6aec0b4330 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:32.588 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:32.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6ae9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:32.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6af348f640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:21:32.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6af348f640 10 monclient: _renew_subs 2024-03-29T11:21:32.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6af348f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:32.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6af348f640 1 -- 172.21.15.157:0/1109265884 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec1e8fd0 con 0x7f6aec0b4330 2024-03-29T11:21:32.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6af348f640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:32.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6af348f640 10 monclient: _renew_subs 2024-03-29T11:21:32.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6af348f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:32.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6af348f640 1 -- 172.21.15.157:0/1109265884 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec1e94b0 con 0x7f6aec0b4330 2024-03-29T11:21:32.589 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.608+0000 7f6af348f640 1 librados: init done 2024-03-29T11:21:32.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.611+0000 7f6af348f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:32.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.611+0000 7f6af348f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:32.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.611+0000 7f6af348f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:32.590 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.611+0000 7f6af348f640 1 -- 172.21.15.157:0/1109265884 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aec075f10 con 0x7f6aec0b4330 2024-03-29T11:21:32.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.614+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6adc001b10 con 0x7f6aec0b4330 2024-03-29T11:21:32.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.614+0000 7f6ae9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:32.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.614+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6adc004f30 con 0x7f6aec0b4330 2024-03-29T11:21:32.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.614+0000 7f6ae9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.614+0000 7f6ae9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:32.591 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.614+0000 7f6ae9ffb640 10 monclient: dump: 2024-03-29T11:21:32.591 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:32.591 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:32.591 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:32.591 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:32.592 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.592 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:32.592 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:32.592 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:32.592 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:32.592 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:32.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.614+0000 7f6aeb7fe640 4 set_mon_vals no callback set 2024-03-29T11:21:32.592 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.615+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f6adc005090 con 0x7f6aec0b4330 2024-03-29T11:21:32.594 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.616+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3066630591 0 0) 0x7f6adc084aa0 con 0x7f6aec0b4330 2024-03-29T11:21:32.594 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.616+0000 7f6ae9ffb640 10 client.4584.objecter ms_dispatch 0x7f6aec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:32.594 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.616+0000 7f6ae9ffb640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:32.594 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.616+0000 7f6ae9ffb640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:32.594 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.616+0000 7f6ae9ffb640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-29T11:21:32.594 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.616+0000 7f6af0a2a640 10 monclient: get_auth_request con 0x7f6ad806dda0 auth_method 0 2024-03-29T11:21:32.594 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.617+0000 7f6ae9ffb640 10 client.4584.objecter ms_handle_connect 0x7f6ad806dda0 2024-03-29T11:21:32.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.625+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6adc0503e0 con 0x7f6aec0b4330 2024-03-29T11:21:32.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.626+0000 7f6ae9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:32.602 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.626+0000 7f6ae9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:32.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.954+0000 7f6af348f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:32.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.954+0000 7f6af348f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:32.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.954+0000 7f6af348f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:32.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.954+0000 7f6af348f640 1 -- 172.21.15.157:0/1109265884 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6aec1df5f0 con 0x7f6aec0b4330 2024-03-29T11:21:32.932 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.955+0000 7f6ae9ffb640 1 -- 172.21.15.157:0/1109265884 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1359762755) 0x7f6adc052310 con 0x7f6aec0b4330 2024-03-29T11:21:32.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.955+0000 7f6ae9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:32.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.955+0000 7f6ae9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:32.933 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:21:32.933 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":10,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:11:36.083691+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:11:36.083691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:32.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.959+0000 7f6af348f640 10 librados: watch_flush enter 2024-03-29T11:21:32.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.959+0000 7f6af348f640 10 librados: watch_flush exit 2024-03-29T11:21:32.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.959+0000 7f6af348f640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-29T11:21:32.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.959+0000 7f6af348f640 10 client.4584.objecter successfully canceled tick 2024-03-29T11:21:32.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.959+0000 7f6af348f640 1 -- 172.21.15.157:0/1109265884 >> v1:172.21.15.157:6816/1747058247 conn(0x7f6ad806dda0 legacy=0x7f6ad8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.959+0000 7f6af348f640 10 monclient: shutdown 2024-03-29T11:21:32.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.959+0000 7f6af348f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:32.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.959+0000 7f6af348f640 1 -- 172.21.15.157:0/1109265884 >> v1:172.21.15.157:6789/0 conn(0x7f6aec0b4330 legacy=0x7f6aec1deb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.961+0000 7f6af348f640 1 -- 172.21.15.157:0/1109265884 shutdown_connections 2024-03-29T11:21:32.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.961+0000 7f6af348f640 1 -- 172.21.15.157:0/1109265884 >> 172.21.15.157:0/1109265884 conn(0x7f6aec066a00 msgr2=0x7f6aec0b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:32.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.961+0000 7f6af348f640 1 -- 172.21.15.157:0/1109265884 shutdown_connections 2024-03-29T11:21:32.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.961+0000 7f6af348f640 1 -- 172.21.15.157:0/1109265884 wait complete. 2024-03-29T11:21:32.938 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:32.961+0000 7f6af348f640 1 librados: shutdown 2024-03-29T11:21:32.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:39.763 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:39.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.936+0000 7feb428c0640 10 monclient: get_monmap_and_config 2024-03-29T11:21:39.913 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.936+0000 7feb428c0640 10 monclient: build_initial_monmap 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.936+0000 7feb428c0640 10 monclient: monmap: 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:21:39.937448+0000 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:21:39.937448+0000 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:21:39.914 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:39.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.937+0000 7feb428c0640 1 Processor -- start 2024-03-29T11:21:39.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.937+0000 7feb428c0640 1 -- start start 2024-03-29T11:21:39.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.937+0000 7feb428c0640 10 monclient: init 2024-03-29T11:21:39.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.938+0000 7feb428c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:39.915 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.938+0000 7feb428c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:39.916 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.938+0000 7feb428c0640 10 monclient(hunting): picked mon.noname-c con 0x7feb3c0af0f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:39.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.938+0000 7feb428c0640 10 monclient(hunting): picked mon.noname-a con 0x7feb3c0af4d0 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:39.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.939+0000 7feb428c0640 10 monclient(hunting): picked mon.noname-b con 0x7feb3c0ab170 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:39.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.940+0000 7feb428c0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c13f010 con 0x7feb3c0af4d0 2024-03-29T11:21:39.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.940+0000 7feb428c0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c0ab6c0 con 0x7feb3c0af0f0 2024-03-29T11:21:39.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.940+0000 7feb428c0640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c0ab9a0 con 0x7feb3c0ab170 2024-03-29T11:21:39.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.940+0000 7feb428c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:39.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.940+0000 7feb428c0640 10 monclient(hunting): authenticate will time out at 1512.136935s 2024-03-29T11:21:39.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.939+0000 7feb3bfff640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7feb3c0af0f0 0x7feb3c13c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:55506/0 (socket says 172.21.15.157:55506) 2024-03-29T11:21:39.917 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.940+0000 7feb3bfff640 1 -- 172.21.15.157:0/2956653524 learned_addr learned my addr 172.21.15.157:0/2956653524 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:39.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.942+0000 7feb3affd640 1 -- 172.21.15.157:0/2956653524 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142359426 0 0) 0x7feb3c0ab6c0 con 0x7feb3c0af0f0 2024-03-29T11:21:39.919 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.942+0000 7feb3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:39.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.942+0000 7feb3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:39.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.942+0000 7feb3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:39.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.942+0000 7feb3affd640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:21:39.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.942+0000 7feb3affd640 1 -- 172.21.15.157:0/2956653524 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb24003520 con 0x7feb3c0af0f0 2024-03-29T11:21:39.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.943+0000 7feb3affd640 1 -- 172.21.15.157:0/2956653524 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2629149819 0 0) 0x7feb24003520 con 0x7feb3c0af0f0 2024-03-29T11:21:39.920 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.943+0000 7feb3affd640 1 -- 172.21.15.157:0/2956653524 >> v1:172.21.15.157:6790/0 conn(0x7feb3c0ab170 legacy=0x7feb3c14a850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:39.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.946+0000 7feb3affd640 1 -- 172.21.15.157:0/2956653524 >> v1:172.21.15.157:6789/0 conn(0x7feb3c0af4d0 legacy=0x7feb3c13f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:39.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.946+0000 7feb3affd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:39.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.946+0000 7feb3affd640 1 monclient: found mon.noname-c 2024-03-29T11:21:39.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.946+0000 7feb3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:39.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:21:39.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 1 -- 172.21.15.157:0/2956653524 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c0abc80 con 0x7feb3c0af0f0 2024-03-29T11:21:39.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 10 monclient: _finish_auth 0 2024-03-29T11:21:39.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:39.925 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 1 -- 172.21.15.157:0/2956653524 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feb34001a10 con 0x7feb3c0af0f0 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb428c0640 5 monclient: authenticate success, global_id 4579 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb428c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 10 monclient: dump: 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:39.926 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 1 -- 172.21.15.157:0/2956653524 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb34001ca0 con 0x7feb3c0af0f0 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 1 -- 172.21.15.157:0/2956653524 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feb34002030 con 0x7feb3c0af0f0 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb3affd640 10 monclient: dump: 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:39.927 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb428c0640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.947+0000 7feb428c0640 4 set_mon_vals no callback set 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.948+0000 7feb428c0640 10 monclient: shutdown 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.948+0000 7feb428c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.948+0000 7feb428c0640 1 -- 172.21.15.157:0/2956653524 >> v1:172.21.15.195:6789/0 conn(0x7feb3c0af0f0 legacy=0x7feb3c13c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.948+0000 7feb428c0640 1 -- 172.21.15.157:0/2956653524 shutdown_connections 2024-03-29T11:21:39.928 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.949+0000 7feb428c0640 1 -- 172.21.15.157:0/2956653524 >> 172.21.15.157:0/2956653524 conn(0x7feb3c0b35e0 msgr2=0x7feb3c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:39.929 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.949+0000 7feb428c0640 1 -- 172.21.15.157:0/2956653524 shutdown_connections 2024-03-29T11:21:39.929 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.949+0000 7feb428c0640 1 -- 172.21.15.157:0/2956653524 wait complete. 2024-03-29T11:21:39.929 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.950+0000 7feb428c0640 10 monclient: build_initial_monmap 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.950+0000 7feb428c0640 10 monclient: monmap: 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.951+0000 7feb428c0640 1 librados: starting msgr at 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.951+0000 7feb428c0640 1 librados: starting objecter 2024-03-29T11:21:39.930 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.951+0000 7feb428c0640 1 Processor -- start 2024-03-29T11:21:39.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.951+0000 7feb428c0640 1 -- start start 2024-03-29T11:21:39.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.951+0000 7feb428c0640 1 librados: setting wanted keys 2024-03-29T11:21:39.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.951+0000 7feb428c0640 1 librados: calling monclient init 2024-03-29T11:21:39.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.951+0000 7feb428c0640 10 monclient: init 2024-03-29T11:21:39.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.952+0000 7feb428c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:39.931 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.952+0000 7feb428c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.952+0000 7feb428c0640 10 monclient(hunting): picked mon.noname-a con 0x7feb3c0ab170 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.952+0000 7feb428c0640 10 monclient(hunting): picked mon.noname-c con 0x7feb3c0af0f0 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.952+0000 7feb428c0640 10 monclient(hunting): picked mon.noname-b con 0x7feb3c13d900 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.952+0000 7feb428c0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c1c85c0 con 0x7feb3c0ab170 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.952+0000 7feb428c0640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c1c8810 con 0x7feb3c13d900 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.952+0000 7feb428c0640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c1c8af0 con 0x7feb3c0af0f0 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.952+0000 7feb428c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.952+0000 7feb428c0640 10 monclient(hunting): authenticate will time out at 1512.149632s 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb40e5d640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7feb3c13d900 0x7feb3c1e5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:56234/0 (socket says 172.21.15.157:56234) 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb40e5d640 1 -- 172.21.15.157:0/3288689216 learned_addr learned my addr 172.21.15.157:0/3288689216 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:39.933 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb38ff9640 10 client.?.objecter ms_handle_connect 0x7feb3c13d900 2024-03-29T11:21:39.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:39.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb38ff9640 10 client.?.objecter ms_handle_connect 0x7feb3c0ab170 2024-03-29T11:21:39.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:39.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034760614 0 0) 0x7feb3c1c8810 con 0x7feb3c13d900 2024-03-29T11:21:39.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb38ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:39.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:39.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:39.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb38ff9640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:21:39.934 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.956+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb28004ee0 con 0x7feb3c13d900 2024-03-29T11:21:39.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.957+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032090543 0 0) 0x7feb3c1c85c0 con 0x7feb3c0ab170 2024-03-29T11:21:39.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.957+0000 7feb38ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:39.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.957+0000 7feb38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:39.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.957+0000 7feb38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:39.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.957+0000 7feb38ff9640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:21:39.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.957+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb3c1c8810 con 0x7feb3c0ab170 2024-03-29T11:21:39.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.957+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 784009407 0 0) 0x7feb3c1c8810 con 0x7feb3c0ab170 2024-03-29T11:21:39.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.957+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb3c1c85c0 con 0x7feb3c0ab170 2024-03-29T11:21:39.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feb340018d0 con 0x7feb3c0ab170 2024-03-29T11:21:39.935 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1361059824 0 0) 0x7feb28004ee0 con 0x7feb3c13d900 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb3c1c8810 con 0x7feb3c13d900 2024-03-29T11:21:39.936 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feb2c0018e0 con 0x7feb3c13d900 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.958+0000 7feb38ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:39.937 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:39.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2873438371 0 0) 0x7feb3c1c85c0 con 0x7feb3c0ab170 2024-03-29T11:21:39.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 >> v1:172.21.15.157:6790/0 conn(0x7feb3c0af0f0 legacy=0x7feb3c13d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:39.940 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 >> v1:172.21.15.195:6789/0 conn(0x7feb3c13d900 legacy=0x7feb3c1e5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 1 monclient: found mon.a 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c13dd40 con 0x7feb3c0ab170 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb428c0640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb34001ab0 con 0x7feb3c0ab170 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb428c0640 10 monclient: _renew_subs 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb428c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb428c0640 1 -- 172.21.15.157:0/3288689216 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3c1c6660 con 0x7feb3c0ab170 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb428c0640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:39.941 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.959+0000 7feb38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.960+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7feb34005110 con 0x7feb3c0ab170 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.960+0000 7feb38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.960+0000 7feb38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.960+0000 7feb38ff9640 10 monclient: dump: 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:39.942 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:39.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.960+0000 7feb428c0640 10 monclient: _renew_subs 2024-03-29T11:21:39.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.960+0000 7feb428c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:39.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.960+0000 7feb428c0640 1 -- 172.21.15.157:0/3288689216 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3c1e77a0 con 0x7feb3c0ab170 2024-03-29T11:21:39.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.960+0000 7feb428c0640 1 librados: init done 2024-03-29T11:21:39.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.961+0000 7feb3a7fc640 4 set_mon_vals no callback set 2024-03-29T11:21:39.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.962+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7feb3401b0e0 con 0x7feb3c0ab170 2024-03-29T11:21:39.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.962+0000 7feb227fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:39.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.963+0000 7feb227fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:39.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.963+0000 7feb227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:39.943 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.963+0000 7feb227fc640 1 -- 172.21.15.157:0/3288689216 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb14005080 con 0x7feb3c0ab170 2024-03-29T11:21:39.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.965+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 417662124 0 0) 0x7feb34084e80 con 0x7feb3c0ab170 2024-03-29T11:21:39.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.965+0000 7feb38ff9640 10 client.4587.objecter ms_dispatch 0x7feb3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:21:39.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.965+0000 7feb38ff9640 3 client.4587.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:21:39.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.965+0000 7feb38ff9640 3 client.4587.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:21:39.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.965+0000 7feb38ff9640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-29T11:21:39.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.965+0000 7feb3b7fe640 10 monclient: get_auth_request con 0x7feb2806dec0 auth_method 0 2024-03-29T11:21:39.944 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.966+0000 7feb38ff9640 10 client.4587.objecter ms_handle_connect 0x7feb2806dec0 2024-03-29T11:21:39.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.971+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb340b28b0 con 0x7feb3c0ab170 2024-03-29T11:21:39.947 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.971+0000 7feb38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:39.948 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:39.971+0000 7feb38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:40.279 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.302+0000 7feb227fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:40.279 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.302+0000 7feb227fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:40.279 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.302+0000 7feb227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:40.279 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.302+0000 7feb227fc640 1 -- 172.21.15.157:0/3288689216 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb14005330 con 0x7feb3c0ab170 2024-03-29T11:21:40.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.306+0000 7feb38ff9640 1 -- 172.21.15.157:0/3288689216 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3435241529) 0x7feb34050760 con 0x7feb3c0ab170 2024-03-29T11:21:40.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.306+0000 7feb38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:40.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.306+0000 7feb38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:40.283 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:21:40.283 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":11,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:21:38.840582+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:21:38.840582+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:40.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.309+0000 7feb227fc640 10 librados: watch_flush enter 2024-03-29T11:21:40.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.310+0000 7feb227fc640 10 librados: watch_flush exit 2024-03-29T11:21:40.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.310+0000 7feb227fc640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-29T11:21:40.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.310+0000 7feb227fc640 10 client.4587.objecter successfully canceled tick 2024-03-29T11:21:40.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.310+0000 7feb227fc640 1 -- 172.21.15.157:0/3288689216 >> v1:172.21.15.157:6816/1747058247 conn(0x7feb2806dec0 legacy=0x7feb28070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:40.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.310+0000 7feb227fc640 10 monclient: shutdown 2024-03-29T11:21:40.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.310+0000 7feb227fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:40.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.310+0000 7feb227fc640 1 -- 172.21.15.157:0/3288689216 >> v1:172.21.15.157:6789/0 conn(0x7feb3c0ab170 legacy=0x7feb3c1c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:40.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.310+0000 7feb227fc640 1 -- 172.21.15.157:0/3288689216 shutdown_connections 2024-03-29T11:21:40.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.310+0000 7feb227fc640 1 -- 172.21.15.157:0/3288689216 >> 172.21.15.157:0/3288689216 conn(0x7feb3c0b35e0 msgr2=0x7feb3c13f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:40.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.311+0000 7feb227fc640 1 -- 172.21.15.157:0/3288689216 shutdown_connections 2024-03-29T11:21:40.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.311+0000 7feb227fc640 1 -- 172.21.15.157:0/3288689216 wait complete. 2024-03-29T11:21:40.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:40.311+0000 7feb227fc640 1 librados: shutdown 2024-03-29T11:21:40.302 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:47.109 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:47.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.281+0000 7fad8d64e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:47.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.281+0000 7fad8d64e640 10 monclient: build_initial_monmap 2024-03-29T11:21:47.258 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.281+0000 7fad8d64e640 10 monclient: monmap: 2024-03-29T11:21:47.258 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:47.258 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:47.258 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:21:47.282691+0000 2024-03-29T11:21:47.258 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:21:47.282691+0000 2024-03-29T11:21:47.258 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:47.259 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:47.259 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:47.259 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:21:47.259 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:21:47.259 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:47.260 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.282+0000 7fad8d64e640 1 Processor -- start 2024-03-29T11:21:47.260 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.282+0000 7fad8d64e640 1 -- start start 2024-03-29T11:21:47.260 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.282+0000 7fad8d64e640 10 monclient: init 2024-03-29T11:21:47.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.283+0000 7fad8d64e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:47.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.283+0000 7fad8d64e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:47.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.283+0000 7fad8d64e640 10 monclient(hunting): picked mon.noname-b con 0x7fad880ae9b0 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:47.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.283+0000 7fad8d64e640 10 monclient(hunting): picked mon.noname-a con 0x7fad88143f50 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:47.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.283+0000 7fad8d64e640 10 monclient(hunting): picked mon.noname-c con 0x7fad880b4890 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:47.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.283+0000 7fad8d64e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad880aed90 con 0x7fad88143f50 2024-03-29T11:21:47.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.283+0000 7fad8d64e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad880af020 con 0x7fad880b4890 2024-03-29T11:21:47.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.283+0000 7fad8d64e640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad880af300 con 0x7fad880ae9b0 2024-03-29T11:21:47.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.283+0000 7fad8d64e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:47.261 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.283+0000 7fad8d64e640 10 monclient(hunting): authenticate will time out at 1519.480441s 2024-03-29T11:21:47.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.285+0000 7fad86ffd640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fad880ae9b0 0x7fad88143840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:57548/0 (socket says 172.21.15.157:57548) 2024-03-29T11:21:47.262 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.285+0000 7fad86ffd640 1 -- 172.21.15.157:0/1038740734 learned_addr learned my addr 172.21.15.157:0/1038740734 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161801337 0 0) 0x7fad880af300 con 0x7fad880ae9b0 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad70003520 con 0x7fad880ae9b0 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038278727 0 0) 0x7fad880aed90 con 0x7fad88143f50 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.265 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:21:47.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.287+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad880af300 con 0x7fad88143f50 2024-03-29T11:21:47.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.288+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 388593671 0 0) 0x7fad880af300 con 0x7fad88143f50 2024-03-29T11:21:47.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.288+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 >> v1:172.21.15.157:6790/0 conn(0x7fad880ae9b0 legacy=0x7fad88143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.266 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.288+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 >> v1:172.21.15.195:6789/0 conn(0x7fad880b4890 legacy=0x7fad88151df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:47.267 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.289+0000 7fad85ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:47.267 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.289+0000 7fad85ffb640 1 monclient: found mon.noname-a 2024-03-29T11:21:47.267 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.289+0000 7fad85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:47.267 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.289+0000 7fad85ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.289+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad880b0710 con 0x7fad88143f50 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.289+0000 7fad85ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.289+0000 7fad85ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.289+0000 7fad85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.289+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fad7c001670 con 0x7fad88143f50 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.289+0000 7fad85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.290+0000 7fad85ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.290+0000 7fad85ffb640 10 monclient: dump: 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.268 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:47.269 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:47.269 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:47.269 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:47.269 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:47.269 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.290+0000 7fad8d64e640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:21:47.269 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.290+0000 7fad8d64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:47.269 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.290+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad7c001850 con 0x7fad88143f50 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.290+0000 7fad85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.290+0000 7fad85ffb640 1 -- 172.21.15.157:0/1038740734 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fad7c001c20 con 0x7fad88143f50 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.290+0000 7fad85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.290+0000 7fad85ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.290+0000 7fad85ffb640 10 monclient: dump: 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:47.270 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:47.271 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:47.271 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.292+0000 7fad8d64e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:47.271 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.292+0000 7fad8d64e640 4 set_mon_vals no callback set 2024-03-29T11:21:47.271 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.293+0000 7fad8d64e640 10 monclient: shutdown 2024-03-29T11:21:47.272 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.293+0000 7fad8d64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:47.272 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.293+0000 7fad8d64e640 1 -- 172.21.15.157:0/1038740734 >> v1:172.21.15.157:6789/0 conn(0x7fad88143f50 legacy=0x7fad880b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.272 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.294+0000 7fad8d64e640 1 -- 172.21.15.157:0/1038740734 shutdown_connections 2024-03-29T11:21:47.272 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.294+0000 7fad8d64e640 1 -- 172.21.15.157:0/1038740734 >> 172.21.15.157:0/1038740734 conn(0x7fad88066a00 msgr2=0x7fad88066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:47.272 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.294+0000 7fad8d64e640 1 -- 172.21.15.157:0/1038740734 shutdown_connections 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.294+0000 7fad8d64e640 1 -- 172.21.15.157:0/1038740734 wait complete. 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.295+0000 7fad8d64e640 10 monclient: build_initial_monmap 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.295+0000 7fad8d64e640 10 monclient: monmap: 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:47.273 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:21:47.274 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:21:47.274 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:47.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.295+0000 7fad8d64e640 1 librados: starting msgr at 2024-03-29T11:21:47.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.295+0000 7fad8d64e640 1 librados: starting objecter 2024-03-29T11:21:47.274 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.296+0000 7fad8d64e640 1 Processor -- start 2024-03-29T11:21:47.275 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.296+0000 7fad8d64e640 1 -- start start 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.296+0000 7fad8d64e640 1 librados: setting wanted keys 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.296+0000 7fad8d64e640 1 librados: calling monclient init 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.296+0000 7fad8d64e640 10 monclient: init 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.297+0000 7fad8d64e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.297+0000 7fad8d64e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.297+0000 7fad8d64e640 10 monclient(hunting): picked mon.noname-b con 0x7fad880ae9b0 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.297+0000 7fad8d64e640 10 monclient(hunting): picked mon.noname-c con 0x7fad880b4890 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.297+0000 7fad8d64e640 10 monclient(hunting): picked mon.noname-a con 0x7fad88143f50 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.297+0000 7fad8d64e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8814a470 con 0x7fad88143f50 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.297+0000 7fad8d64e640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad88070fb0 con 0x7fad880ae9b0 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.297+0000 7fad8d64e640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8806ee20 con 0x7fad880b4890 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.297+0000 7fad8d64e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:47.276 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.297+0000 7fad8d64e640 10 monclient(hunting): authenticate will time out at 1519.494462s 2024-03-29T11:21:47.277 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.300+0000 7fad877fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fad88143f50 0x7fad881e7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:59915/0 (socket says 172.21.15.157:59915) 2024-03-29T11:21:47.277 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.301+0000 7fad877fe640 1 -- 172.21.15.157:0/934417569 learned_addr learned my addr 172.21.15.157:0/934417569 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:47.277 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.301+0000 7fad6f7fe640 10 client.?.objecter ms_handle_connect 0x7fad88143f50 2024-03-29T11:21:47.278 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.301+0000 7fad6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:47.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 10 client.?.objecter ms_handle_connect 0x7fad880b4890 2024-03-29T11:21:47.282 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066559798 0 0) 0x7fad8814a470 con 0x7fad88143f50 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad74004eb0 con 0x7fad88143f50 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 953483956 0 0) 0x7fad74004eb0 con 0x7fad88143f50 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad74005300 con 0x7fad88143f50 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fad780017b0 con 0x7fad88143f50 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.302+0000 7fad6f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:47.283 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 10 client.?.objecter ms_handle_connect 0x7fad880ae9b0 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370557817 0 0) 0x7fad88070fb0 con 0x7fad880ae9b0 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.284 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad74005850 con 0x7fad880ae9b0 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3022956112 0 0) 0x7fad74005300 con 0x7fad88143f50 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.303+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 >> v1:172.21.15.157:6790/0 conn(0x7fad880b4890 legacy=0x7fad8806f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 >> v1:172.21.15.195:6789/0 conn(0x7fad880ae9b0 legacy=0x7fad880716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad6f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad6f7fe640 1 monclient: found mon.a 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad6f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8806de90 con 0x7fad88143f50 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad6f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad6f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad8d64e640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:21:47.285 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad8d64e640 10 monclient: _renew_subs 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad8d64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad8d64e640 1 -- 172.21.15.157:0/934417569 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad881e7150 con 0x7fad88143f50 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad8d64e640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad8d64e640 10 monclient: _renew_subs 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad8d64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad8d64e640 1 -- 172.21.15.157:0/934417569 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8806fc90 con 0x7fad88143f50 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.304+0000 7fad8d64e640 1 librados: init done 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.305+0000 7fad8d64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.305+0000 7fad8d64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.305+0000 7fad8d64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:47.286 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.305+0000 7fad8d64e640 1 -- 172.21.15.157:0/934417569 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad880688a0 con 0x7fad88143f50 2024-03-29T11:21:47.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.307+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad78001d40 con 0x7fad88143f50 2024-03-29T11:21:47.287 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.307+0000 7fad6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.307+0000 7fad857fa640 4 set_mon_vals no callback set 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.307+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fad780017b0 con 0x7fad88143f50 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.307+0000 7fad6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.307+0000 7fad6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.307+0000 7fad6f7fe640 10 monclient: dump: 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:47.288 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:47.289 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:47.289 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:47.289 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.309+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fad7801a770 con 0x7fad88143f50 2024-03-29T11:21:47.289 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.310+0000 7fad86ffd640 10 monclient: get_auth_request con 0x7fad7406e350 auth_method 0 2024-03-29T11:21:47.289 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.310+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 417662124 0 0) 0x7fad780542a0 con 0x7fad88143f50 2024-03-29T11:21:47.289 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.310+0000 7fad6f7fe640 10 client.4593.objecter ms_dispatch 0x7fad88000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:21:47.289 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.310+0000 7fad6f7fe640 3 client.4593.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:21:47.289 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.310+0000 7fad6f7fe640 3 client.4593.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:21:47.289 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.310+0000 7fad6f7fe640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-29T11:21:47.289 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.311+0000 7fad6f7fe640 10 client.4593.objecter ms_handle_connect 0x7fad7406e350 2024-03-29T11:21:47.293 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.316+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad78050540 con 0x7fad88143f50 2024-03-29T11:21:47.293 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.316+0000 7fad6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:47.293 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.316+0000 7fad6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:47.614 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.638+0000 7fad8d64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:47.614 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.638+0000 7fad8d64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:47.614 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.638+0000 7fad8d64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:21:47.615 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.638+0000 7fad8d64e640 1 -- 172.21.15.157:0/934417569 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad8806f060 con 0x7fad88143f50 2024-03-29T11:21:47.615 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.639+0000 7fad6f7fe640 1 -- 172.21.15.157:0/934417569 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3435241529) 0x7fad78050540 con 0x7fad88143f50 2024-03-29T11:21:47.615 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.639+0000 7fad6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:47.616 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.639+0000 7fad6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:47.616 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:21:47.616 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":11,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:21:38.840582+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:21:38.840582+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:47.619 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.643+0000 7fad8d64e640 10 librados: watch_flush enter 2024-03-29T11:21:47.620 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.643+0000 7fad8d64e640 10 librados: watch_flush exit 2024-03-29T11:21:47.620 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.643+0000 7fad8d64e640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-29T11:21:47.620 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.643+0000 7fad8d64e640 10 client.4593.objecter successfully canceled tick 2024-03-29T11:21:47.621 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.644+0000 7fad8d64e640 1 -- 172.21.15.157:0/934417569 >> v1:172.21.15.157:6816/1747058247 conn(0x7fad7406e350 legacy=0x7fad74070810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.621 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.644+0000 7fad8d64e640 10 monclient: shutdown 2024-03-29T11:21:47.621 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.644+0000 7fad8d64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:47.622 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.644+0000 7fad8d64e640 1 -- 172.21.15.157:0/934417569 >> v1:172.21.15.157:6789/0 conn(0x7fad88143f50 legacy=0x7fad881e7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.622 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.644+0000 7fad8d64e640 1 -- 172.21.15.157:0/934417569 shutdown_connections 2024-03-29T11:21:47.622 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.644+0000 7fad8d64e640 1 -- 172.21.15.157:0/934417569 >> 172.21.15.157:0/934417569 conn(0x7fad88066a00 msgr2=0x7fad880b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:47.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.645+0000 7fad8d64e640 1 -- 172.21.15.157:0/934417569 shutdown_connections 2024-03-29T11:21:47.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.646+0000 7fad8d64e640 1 -- 172.21.15.157:0/934417569 wait complete. 2024-03-29T11:21:47.623 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:47.646+0000 7fad8d64e640 1 librados: shutdown 2024-03-29T11:21:47.640 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:54.446 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:54.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.647+0000 7f35dae6d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:54.626 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.647+0000 7f35dae6d640 10 monclient: build_initial_monmap 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.647+0000 7f35dae6d640 10 monclient: monmap: 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:21:54.648424+0000 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:21:54.648424+0000 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.649+0000 7f35dae6d640 1 Processor -- start 2024-03-29T11:21:54.627 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.650+0000 7f35dae6d640 1 -- start start 2024-03-29T11:21:54.628 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.650+0000 7f35dae6d640 10 monclient: init 2024-03-29T11:21:54.628 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.650+0000 7f35dae6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:54.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.650+0000 7f35dae6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:54.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.650+0000 7f35dae6d640 10 monclient(hunting): picked mon.noname-c con 0x7f35cc006410 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:54.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.651+0000 7f35dae6d640 10 monclient(hunting): picked mon.noname-a con 0x7f35cc02a640 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:54.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.651+0000 7f35dae6d640 10 monclient(hunting): picked mon.noname-b con 0x7f35cc02b370 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:54.629 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.651+0000 7f35dae6d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc02a120 con 0x7f35cc02a640 2024-03-29T11:21:54.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.651+0000 7f35dae6d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc0c75b0 con 0x7f35cc006410 2024-03-29T11:21:54.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.651+0000 7f35dae6d640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc033110 con 0x7f35cc02b370 2024-03-29T11:21:54.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.651+0000 7f35dae6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:54.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.651+0000 7f35dae6d640 10 monclient(hunting): authenticate will time out at 1526.847927s 2024-03-29T11:21:54.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.651+0000 7f35da66c640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f35cc02b370 0x7f35cc0320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:37306/0 (socket says 172.21.15.157:37306) 2024-03-29T11:21:54.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.651+0000 7f35da66c640 1 -- 172.21.15.157:0/2219028868 learned_addr learned my addr 172.21.15.157:0/2219028868 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:54.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.654+0000 7f35d8e69640 1 -- 172.21.15.157:0/2219028868 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518732126 0 0) 0x7f35cc033110 con 0x7f35cc02b370 2024-03-29T11:21:54.631 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.654+0000 7f35d8e69640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.632 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.654+0000 7f35d8e69640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.632 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.654+0000 7f35d8e69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.632 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.654+0000 7f35d8e69640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:21:54.632 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.654+0000 7f35d8e69640 1 -- 172.21.15.157:0/2219028868 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c0003520 con 0x7f35cc02b370 2024-03-29T11:21:54.635 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 1 -- 172.21.15.157:0/2219028868 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1709002151 0 0) 0x7f35c0003520 con 0x7f35cc02b370 2024-03-29T11:21:54.635 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 1 -- 172.21.15.157:0/2219028868 >> v1:172.21.15.195:6789/0 conn(0x7f35cc006410 legacy=0x7f35cc0345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.635 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 1 -- 172.21.15.157:0/2219028868 >> v1:172.21.15.157:6789/0 conn(0x7f35cc02a640 legacy=0x7f35cc0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.635 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 10 monclient: _finish_hunting 0 2024-03-29T11:21:54.635 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 1 monclient: found mon.noname-b 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 1 -- 172.21.15.157:0/2219028868 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc0332a0 con 0x7f35cc02b370 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 10 monclient: _finish_auth 0 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 10 monclient: _check_auth_tickets 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 1 -- 172.21.15.157:0/2219028868 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f35c4001670 con 0x7f35cc02b370 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.655+0000 7f35d8e69640 10 monclient: dump: 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:54.636 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.656+0000 7f35dae6d640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.656+0000 7f35dae6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.657+0000 7f35d8e69640 1 -- 172.21.15.157:0/2219028868 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35c4001890 con 0x7f35cc02b370 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.657+0000 7f35d8e69640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.657+0000 7f35d8e69640 1 -- 172.21.15.157:0/2219028868 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f35c4001c20 con 0x7f35cc02b370 2024-03-29T11:21:54.637 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.657+0000 7f35d8e69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.657+0000 7f35d8e69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.657+0000 7f35d8e69640 10 monclient: dump: 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:54.638 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:54.639 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.657+0000 7f35dae6d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:54.639 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.657+0000 7f35dae6d640 4 set_mon_vals no callback set 2024-03-29T11:21:54.639 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.658+0000 7f35dae6d640 10 monclient: shutdown 2024-03-29T11:21:54.639 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.658+0000 7f35dae6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:54.639 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.658+0000 7f35dae6d640 1 -- 172.21.15.157:0/2219028868 >> v1:172.21.15.157:6790/0 conn(0x7f35cc02b370 legacy=0x7f35cc0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.659+0000 7f35dae6d640 1 -- 172.21.15.157:0/2219028868 shutdown_connections 2024-03-29T11:21:54.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.659+0000 7f35dae6d640 1 -- 172.21.15.157:0/2219028868 >> 172.21.15.157:0/2219028868 conn(0x7f35cc0058f0 msgr2=0x7f35cc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:54.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.660+0000 7f35dae6d640 1 -- 172.21.15.157:0/2219028868 shutdown_connections 2024-03-29T11:21:54.640 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.660+0000 7f35dae6d640 1 -- 172.21.15.157:0/2219028868 wait complete. 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.661+0000 7f35dae6d640 10 monclient: build_initial_monmap 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.661+0000 7f35dae6d640 10 monclient: monmap: 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.661+0000 7f35dae6d640 1 librados: starting msgr at 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.661+0000 7f35dae6d640 1 librados: starting objecter 2024-03-29T11:21:54.641 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.662+0000 7f35dae6d640 1 Processor -- start 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.662+0000 7f35dae6d640 1 -- start start 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.662+0000 7f35dae6d640 1 librados: setting wanted keys 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.662+0000 7f35dae6d640 1 librados: calling monclient init 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.662+0000 7f35dae6d640 10 monclient: init 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.662+0000 7f35dae6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.663+0000 7f35dae6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.663+0000 7f35dae6d640 10 monclient(hunting): picked mon.noname-c con 0x7f35cc006410 addr v1:172.21.15.157:6790/0 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.663+0000 7f35dae6d640 10 monclient(hunting): picked mon.noname-b con 0x7f35cc02a640 addr v1:172.21.15.195:6789/0 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.663+0000 7f35dae6d640 10 monclient(hunting): picked mon.noname-a con 0x7f35cc02b370 addr v1:172.21.15.157:6789/0 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.663+0000 7f35dae6d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc16d250 con 0x7f35cc02b370 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.663+0000 7f35dae6d640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc0bd840 con 0x7f35cc02a640 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.663+0000 7f35dae6d640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc169c80 con 0x7f35cc006410 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.663+0000 7f35dae6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:54.642 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.663+0000 7f35dae6d640 10 monclient(hunting): authenticate will time out at 1526.859953s 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.666+0000 7f35d966a640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f35cc02a640 0x7f35cc169570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:50773/0 (socket says 172.21.15.157:50773) 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.666+0000 7f35d966a640 1 -- 172.21.15.157:0/1070314211 learned_addr learned my addr 172.21.15.157:0/1070314211 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.666+0000 7f35ca7fc640 10 client.?.objecter ms_handle_connect 0x7f35cc02a640 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.666+0000 7f35ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609375428 0 0) 0x7f35cc0bd840 con 0x7f35cc02a640 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b8004eb0 con 0x7f35cc02a640 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 10 client.?.objecter ms_handle_connect 0x7f35cc02b370 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:54.648 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2317297512 0 0) 0x7f35b8004eb0 con 0x7f35cc02a640 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35b8005300 con 0x7f35cc02a640 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f35d404fdd0 con 0x7f35cc02a640 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.667+0000 7f35ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:54.649 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012210457 0 0) 0x7f35cc16d250 con 0x7f35cc02b370 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b8004eb0 con 0x7f35cc02b370 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 client.?.objecter ms_handle_connect 0x7f35cc006410 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.1 v1:172.21.15.195:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4011239188 0 0) 0x7f35b8005300 con 0x7f35cc02a640 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 >> v1:172.21.15.157:6790/0 conn(0x7f35cc006410 legacy=0x7f35cc0bd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.650 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 >> v1:172.21.15.157:6789/0 conn(0x7f35cc02b370 legacy=0x7f35cc166680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 1 monclient: found mon.b 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc16d770 con 0x7f35cc02a640 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.668+0000 7f35ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35dae6d640 5 monclient: authenticate success, global_id 4591 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35dae6d640 10 monclient: _renew_subs 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35dae6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35dae6d640 1 -- 172.21.15.157:0/1070314211 --> v1:172.21.15.195:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35cc16d9c0 con 0x7f35cc02a640 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35dae6d640 10 client.4591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35dae6d640 10 monclient: _renew_subs 2024-03-29T11:21:54.651 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35dae6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35dae6d640 1 -- 172.21.15.157:0/1070314211 --> v1:172.21.15.195:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35cc166f70 con 0x7f35cc02a640 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35dae6d640 1 librados: init done 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.1 v1:172.21.15.195:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35d404f400 con 0x7f35cc02a640 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.669+0000 7f35ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.670+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.1 v1:172.21.15.195:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f35d4075f10 con 0x7f35cc02a640 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.670+0000 7f35ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.670+0000 7f35ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.670+0000 7f35ca7fc640 10 monclient: dump: 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.652 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:21:54.653 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:21:54.653 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:21:54.653 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:21:54.653 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:21:54.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.670+0000 7f35cb7fe640 4 set_mon_vals no callback set 2024-03-29T11:21:54.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.670+0000 7f35dae6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:54.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.671+0000 7f35dae6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:54.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.671+0000 7f35dae6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:21:54.653 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.671+0000 7f35dae6d640 1 -- 172.21.15.157:0/1070314211 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc005080 con 0x7f35cc02a640 2024-03-29T11:21:54.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.672+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.1 v1:172.21.15.195:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f35d4067bf0 con 0x7f35cc02a640 2024-03-29T11:21:54.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.673+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.1 v1:172.21.15.195:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 417662124 0 0) 0x7f35d40cf1b0 con 0x7f35cc02a640 2024-03-29T11:21:54.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.673+0000 7f35ca7fc640 10 client.4591.objecter ms_dispatch 0x7f35d4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:21:54.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.673+0000 7f35ca7fc640 3 client.4591.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:21:54.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.673+0000 7f35ca7fc640 3 client.4591.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:21:54.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.673+0000 7f35ca7fc640 20 client.4591.objecter dump_active .. 0 homeless 2024-03-29T11:21:54.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.674+0000 7f35d9e6b640 10 monclient: get_auth_request con 0x7f35b806dfc0 auth_method 0 2024-03-29T11:21:54.654 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.674+0000 7f35ca7fc640 10 client.4591.objecter ms_handle_connect 0x7f35b806dfc0 2024-03-29T11:21:54.655 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.678+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.1 v1:172.21.15.195:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35d40cb450 con 0x7f35cc02a640 2024-03-29T11:21:54.655 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.678+0000 7f35ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:54.655 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:54.678+0000 7f35ca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:54.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.015+0000 7f35dae6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:54.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.015+0000 7f35dae6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:54.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.015+0000 7f35dae6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.195:6789/0 2024-03-29T11:21:54.992 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.015+0000 7f35dae6d640 1 -- 172.21.15.157:0/1070314211 --> v1:172.21.15.195:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35bc005330 con 0x7f35cc02a640 2024-03-29T11:21:54.995 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.019+0000 7f35ca7fc640 1 -- 172.21.15.157:0/1070314211 <== mon.1 v1:172.21.15.195:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3435241529) 0x7f35d40cb450 con 0x7f35cc02a640 2024-03-29T11:21:54.995 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.019+0000 7f35ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:54.995 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.019+0000 7f35ca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:54.996 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:21:54.996 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":11,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:21:38.840582+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:21:38.840582+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:55.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.023+0000 7f35dae6d640 10 librados: watch_flush enter 2024-03-29T11:21:55.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.024+0000 7f35dae6d640 10 librados: watch_flush exit 2024-03-29T11:21:55.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.024+0000 7f35dae6d640 20 client.4591.objecter shutdown clearing up homeless session... 2024-03-29T11:21:55.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.024+0000 7f35dae6d640 10 client.4591.objecter successfully canceled tick 2024-03-29T11:21:55.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.025+0000 7f35dae6d640 1 -- 172.21.15.157:0/1070314211 >> v1:172.21.15.157:6816/1747058247 conn(0x7f35b806dfc0 legacy=0x7f35b8070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.025+0000 7f35dae6d640 10 monclient: shutdown 2024-03-29T11:21:55.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.025+0000 7f35dae6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:55.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.025+0000 7f35dae6d640 1 -- 172.21.15.157:0/1070314211 >> v1:172.21.15.195:6789/0 conn(0x7f35cc02a640 legacy=0x7f35cc169570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.025+0000 7f35dae6d640 1 -- 172.21.15.157:0/1070314211 shutdown_connections 2024-03-29T11:21:55.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.025+0000 7f35dae6d640 1 -- 172.21.15.157:0/1070314211 >> 172.21.15.157:0/1070314211 conn(0x7f35cc0058f0 msgr2=0x7f35cc0379a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:55.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.026+0000 7f35dae6d640 1 -- 172.21.15.157:0/1070314211 shutdown_connections 2024-03-29T11:21:55.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.026+0000 7f35dae6d640 1 -- 172.21.15.157:0/1070314211 wait complete. 2024-03-29T11:21:55.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:21:55.026+0000 7f35dae6d640 1 librados: shutdown 2024-03-29T11:21:55.020 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:01.826 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:01.979 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.002+0000 7f3588806640 10 monclient: get_monmap_and_config 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.003+0000 7f3588806640 10 monclient: build_initial_monmap 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.003+0000 7f3588806640 10 monclient: monmap: 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:22:02.004272+0000 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:22:02.004272+0000 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:22:01.980 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:01.981 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.004+0000 7f3588806640 1 Processor -- start 2024-03-29T11:22:01.981 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.004+0000 7f3588806640 1 -- start start 2024-03-29T11:22:01.981 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.004+0000 7f3588806640 10 monclient: init 2024-03-29T11:22:01.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.005+0000 7f3588806640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:01.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.005+0000 7f3588806640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:01.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.005+0000 7f3588806640 10 monclient(hunting): picked mon.noname-a con 0x7f35841457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:22:01.985 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.005+0000 7f3588806640 10 monclient(hunting): picked mon.noname-b con 0x7f358413f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.005+0000 7f358259c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f35841457f0 0x7f358413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:33838/0 (socket says 172.21.15.157:33838) 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.005+0000 7f358259c640 1 -- 172.21.15.157:0/4223619203 learned_addr learned my addr 172.21.15.157:0/4223619203 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.005+0000 7f3588806640 10 monclient(hunting): picked mon.noname-c con 0x7f35840a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f3588806640 1 -- 172.21.15.157:0/4223619203 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35840aaef0 con 0x7f35841457f0 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f3588806640 1 -- 172.21.15.157:0/4223619203 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3584145bd0 con 0x7f35840a9d70 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f3588806640 1 -- 172.21.15.157:0/4223619203 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3584145eb0 con 0x7f358413f650 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f3588806640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f3588806640 10 monclient(hunting): authenticate will time out at 1534.203193s 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338609016 0 0) 0x7f35840aaef0 con 0x7f35841457f0 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:01.986 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f356c003520 con 0x7f35841457f0 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121294520 0 0) 0x7f3584145bd0 con 0x7f35840a9d70 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35840aaef0 con 0x7f35840a9d70 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828767330 0 0) 0x7f3584145eb0 con 0x7f358413f650 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:01.987 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.006+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f356c004a30 con 0x7f358413f650 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2532919111 0 0) 0x7f35840aaef0 con 0x7f35840a9d70 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 >> v1:172.21.15.157:6790/0 conn(0x7f358413f650 legacy=0x7f35840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 >> v1:172.21.15.157:6789/0 conn(0x7f35841457f0 legacy=0x7f358413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 1 monclient: found mon.noname-c 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3584146340 con 0x7f35840a9d70 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: _finish_auth 0 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f357c001970 con 0x7f35840a9d70 2024-03-29T11:22:01.988 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: dump: 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f3588806640 5 monclient: authenticate success, global_id 4594 2024-03-29T11:22:01.989 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f3588806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f357c001b50 con 0x7f35840a9d70 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 1 -- 172.21.15.157:0/4223619203 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f357c001fa0 con 0x7f35840a9d70 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f358159a640 10 monclient: dump: 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:01.990 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f3588806640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.007+0000 7f3588806640 4 set_mon_vals no callback set 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.008+0000 7f3588806640 10 monclient: shutdown 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.008+0000 7f3588806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.008+0000 7f3588806640 1 -- 172.21.15.157:0/4223619203 >> v1:172.21.15.195:6789/0 conn(0x7f35840a9d70 legacy=0x7f358414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.009+0000 7f3588806640 1 -- 172.21.15.157:0/4223619203 shutdown_connections 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.009+0000 7f3588806640 1 -- 172.21.15.157:0/4223619203 >> 172.21.15.157:0/4223619203 conn(0x7f3584066a00 msgr2=0x7f3584066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.009+0000 7f3588806640 1 -- 172.21.15.157:0/4223619203 shutdown_connections 2024-03-29T11:22:01.991 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.009+0000 7f3588806640 1 -- 172.21.15.157:0/4223619203 wait complete. 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.010+0000 7f3588806640 10 monclient: build_initial_monmap 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.010+0000 7f3588806640 10 monclient: monmap: 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:22:01.998 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.011+0000 7f3588806640 1 librados: starting msgr at 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.011+0000 7f3588806640 1 librados: starting objecter 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.011+0000 7f3588806640 1 Processor -- start 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.011+0000 7f3588806640 1 -- start start 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.011+0000 7f3588806640 1 librados: setting wanted keys 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.011+0000 7f3588806640 1 librados: calling monclient init 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.011+0000 7f3588806640 10 monclient: init 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3588806640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3588806640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3588806640 10 monclient(hunting): picked mon.noname-a con 0x7f35840a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3588806640 10 monclient(hunting): picked mon.noname-c con 0x7f358413f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3581d9b640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f358413f650 0x7f3584137430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:54812/0 (socket says 172.21.15.157:54812) 2024-03-29T11:22:01.999 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f358259c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f35840a9d70 0x7f3584141b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:34152/0 (socket says 172.21.15.157:34152) 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3588806640 10 monclient(hunting): picked mon.noname-b con 0x7f35841457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3588806640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3584138250 con 0x7f35840a9d70 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3584141090 con 0x7f35841457f0 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3581d9b640 1 -- 172.21.15.157:0/3626144058 learned_addr learned my addr 172.21.15.157:0/3626144058 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3584141370 con 0x7f358413f650 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3588806640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.013+0000 7f3588806640 10 monclient(hunting): authenticate will time out at 1534.210634s 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 10 client.?.objecter ms_handle_connect 0x7f358413f650 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 10 client.?.objecter ms_handle_connect 0x7f35840a9d70 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914105774 0 0) 0x7f3584141370 con 0x7f358413f650 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:02.000 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3564004eb0 con 0x7f358413f650 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 10 client.?.objecter ms_handle_connect 0x7f35841457f0 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.014+0000 7f3572ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957139171 0 0) 0x7f3584141090 con 0x7f35841457f0 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): my global_id is 4597 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3584141370 con 0x7f35841457f0 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170289654 0 0) 0x7f3584138250 con 0x7f35840a9d70 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:02.001 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): my global_id is 4605 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3584141090 con 0x7f35840a9d70 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3464391811 0 0) 0x7f3564004eb0 con 0x7f358413f650 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3584138250 con 0x7f358413f650 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3578001a70 con 0x7f358413f650 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:02.002 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.015+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1926547458 0 0) 0x7f3584141090 con 0x7f35840a9d70 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3564004eb0 con 0x7f35840a9d70 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f35740016b0 con 0x7f35840a9d70 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:02.003 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3701042976 0 0) 0x7f3584141370 con 0x7f35841457f0 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.195:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3584141090 con 0x7f35841457f0 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2559507724 0 0) 0x7f3584138250 con 0x7f358413f650 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 >> v1:172.21.15.195:6789/0 conn(0x7f35841457f0 legacy=0x7f3584137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 >> v1:172.21.15.157:6789/0 conn(0x7f35840a9d70 legacy=0x7f3584141b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 1 monclient: found mon.c 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.016+0000 7f3572ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3584141650 con 0x7f358413f650 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3572ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3572ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:02.004 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3588806640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3588806640 10 monclient: _renew_subs 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3588806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358413ff20 con 0x7f358413f650 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3588806640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3588806640 10 monclient: _renew_subs 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3588806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3584145360 con 0x7f358413f650 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3578001cd0 con 0x7f358413f650 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3588806640 1 librados: init done 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3572ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3580d99640 4 set_mon_vals no callback set 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3578003a40 con 0x7f358413f650 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3572ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:02.005 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.017+0000 7f3572ffd640 10 monclient: dump: 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.019+0000 7f3588806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.019+0000 7f3588806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.019+0000 7f3588806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:02.006 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.019+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358413fd20 con 0x7f358413f650 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.020+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f3578003ba0 con 0x7f358413f650 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.020+0000 7f358259c640 10 monclient: get_auth_request con 0x7f356406e080 auth_method 0 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.021+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 417662124 0 0) 0x7f3578084a10 con 0x7f358413f650 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.021+0000 7f3572ffd640 10 client.4619.objecter ms_dispatch 0x7f3584000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.021+0000 7f3572ffd640 3 client.4619.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.021+0000 7f3572ffd640 3 client.4619.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.021+0000 7f3572ffd640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.021+0000 7f3572ffd640 10 client.4619.objecter ms_handle_connect 0x7f356406e080 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.029+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3578050350 con 0x7f358413f650 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.030+0000 7f3572ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:02.007 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.030+0000 7f3572ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:02.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.394+0000 7f3588806640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:02.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.394+0000 7f3588806640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:02.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.394+0000 7f3588806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:02.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.394+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3584075f10 con 0x7f358413f650 2024-03-29T11:22:02.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.395+0000 7f3572ffd640 1 -- 172.21.15.157:0/3626144058 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3435241529) 0x7f35780840c0 con 0x7f358413f650 2024-03-29T11:22:02.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.395+0000 7f3572ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:02.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.395+0000 7f3572ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:02.372 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:22:02.372 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":11,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:21:38.840582+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:21:38.840582+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:02.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.399+0000 7f3588806640 10 librados: watch_flush enter 2024-03-29T11:22:02.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.399+0000 7f3588806640 10 librados: watch_flush exit 2024-03-29T11:22:02.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.399+0000 7f3588806640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-29T11:22:02.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.399+0000 7f3588806640 10 client.4619.objecter successfully canceled tick 2024-03-29T11:22:02.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.399+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 >> v1:172.21.15.157:6816/1747058247 conn(0x7f356406e080 legacy=0x7f3564070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.399+0000 7f3588806640 10 monclient: shutdown 2024-03-29T11:22:02.376 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.399+0000 7f3588806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:02.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.399+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 >> v1:172.21.15.157:6790/0 conn(0x7f358413f650 legacy=0x7f3584137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.399+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 shutdown_connections 2024-03-29T11:22:02.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.399+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 >> 172.21.15.157:0/3626144058 conn(0x7f3584066a00 msgr2=0x7f35840b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:02.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.400+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 shutdown_connections 2024-03-29T11:22:02.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.400+0000 7f3588806640 1 -- 172.21.15.157:0/3626144058 wait complete. 2024-03-29T11:22:02.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:02.400+0000 7f3588806640 1 librados: shutdown 2024-03-29T11:22:02.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:09.200 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:09.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.370+0000 7f3e1617b640 10 monclient: get_monmap_and_config 2024-03-29T11:22:09.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.370+0000 7f3e1617b640 10 monclient: build_initial_monmap 2024-03-29T11:22:09.347 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.370+0000 7f3e1617b640 10 monclient: monmap: 2024-03-29T11:22:09.347 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:22:09.348 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:09.348 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:22:09.372050+0000 2024-03-29T11:22:09.348 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:22:09.372050+0000 2024-03-29T11:22:09.348 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:09.348 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:09.348 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:22:09.348 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:22:09.348 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:22:09.348 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:09.349 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.372+0000 7f3e1617b640 1 Processor -- start 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.373+0000 7f3e1617b640 1 -- start start 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.373+0000 7f3e1617b640 10 monclient: init 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.374+0000 7f3e1617b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.374+0000 7f3e1617b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.374+0000 7f3e1617b640 10 monclient(hunting): picked mon.noname-c con 0x7f3e101457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.374+0000 7f3e1617b640 10 monclient(hunting): picked mon.noname-a con 0x7f3e1013f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.374+0000 7f3e1617b640 10 monclient(hunting): picked mon.noname-b con 0x7f3e100a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.374+0000 7f3e1617b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e100aaef0 con 0x7f3e1013f650 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.374+0000 7f3e1617b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e10145bd0 con 0x7f3e101457f0 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.374+0000 7f3e1617b640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e10145eb0 con 0x7f3e100a9d70 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.374+0000 7f3e1617b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:09.351 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.374+0000 7f3e1617b640 10 monclient(hunting): authenticate will time out at 1541.571730s 2024-03-29T11:22:09.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.380+0000 7f3e0ffff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f3e100a9d70 0x7f3e1014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:39849/0 (socket says 172.21.15.157:39849) 2024-03-29T11:22:09.356 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.380+0000 7f3e0ffff640 1 -- 172.21.15.157:0/965724972 learned_addr learned my addr 172.21.15.157:0/965724972 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:22:09.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.380+0000 7f3e0e7fc640 1 -- 172.21.15.157:0/965724972 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407579947 0 0) 0x7f3e10145eb0 con 0x7f3e100a9d70 2024-03-29T11:22:09.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.380+0000 7f3e0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.380+0000 7f3e0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.380+0000 7f3e0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.380+0000 7f3e0e7fc640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:22:09.357 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.380+0000 7f3e0e7fc640 1 -- 172.21.15.157:0/965724972 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df8003520 con 0x7f3e100a9d70 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 1 -- 172.21.15.157:0/965724972 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3588079065 0 0) 0x7f3df8003520 con 0x7f3e100a9d70 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 1 -- 172.21.15.157:0/965724972 >> v1:172.21.15.195:6789/0 conn(0x7f3e101457f0 legacy=0x7f3e1013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 1 -- 172.21.15.157:0/965724972 >> v1:172.21.15.157:6789/0 conn(0x7f3e1013f650 legacy=0x7f3e100b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 1 -- 172.21.15.157:0/965724972 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e10146340 con 0x7f3e100a9d70 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 1 -- 172.21.15.157:0/965724972 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3e08001a10 con 0x7f3e100a9d70 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.358 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: dump: 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 1 -- 172.21.15.157:0/965724972 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e08001be0 con 0x7f3e100a9d70 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:09.359 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 1 -- 172.21.15.157:0/965724972 <== mon.2 v1:172.21.15.157:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3e08002030 con 0x7f3e100a9d70 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e0e7fc640 10 monclient: dump: 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:09.360 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:09.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e1617b640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:22:09.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e1617b640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:09.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.381+0000 7f3e1617b640 4 set_mon_vals no callback set 2024-03-29T11:22:09.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.382+0000 7f3e1617b640 10 monclient: shutdown 2024-03-29T11:22:09.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.382+0000 7f3e1617b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:09.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.382+0000 7f3e1617b640 1 -- 172.21.15.157:0/965724972 >> v1:172.21.15.157:6790/0 conn(0x7f3e100a9d70 legacy=0x7f3e1014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.382+0000 7f3e1617b640 1 -- 172.21.15.157:0/965724972 shutdown_connections 2024-03-29T11:22:09.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.382+0000 7f3e1617b640 1 -- 172.21.15.157:0/965724972 >> 172.21.15.157:0/965724972 conn(0x7f3e10066a00 msgr2=0x7f3e10066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:09.361 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.385+0000 7f3e1617b640 1 -- 172.21.15.157:0/965724972 shutdown_connections 2024-03-29T11:22:09.362 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.386+0000 7f3e1617b640 1 -- 172.21.15.157:0/965724972 wait complete. 2024-03-29T11:22:09.365 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.387+0000 7f3e1617b640 10 monclient: build_initial_monmap 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.387+0000 7f3e1617b640 10 monclient: monmap: 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:09.366 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.387+0000 7f3e1617b640 1 librados: starting msgr at 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.387+0000 7f3e1617b640 1 librados: starting objecter 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.387+0000 7f3e1617b640 1 Processor -- start 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 1 -- start start 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 1 librados: setting wanted keys 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 1 librados: calling monclient init 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 10 monclient: init 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 10 monclient(hunting): picked mon.noname-c con 0x7f3e100a9d70 addr v1:172.21.15.157:6790/0 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 10 monclient(hunting): picked mon.noname-a con 0x7f3e1013f650 addr v1:172.21.15.157:6789/0 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 10 monclient(hunting): picked mon.noname-b con 0x7f3e101457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e10140600 con 0x7f3e1013f650 2024-03-29T11:22:09.367 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e101408e0 con 0x7f3e101457f0 2024-03-29T11:22:09.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e10140bc0 con 0x7f3e100a9d70 2024-03-29T11:22:09.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:09.368 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.388+0000 7f3e1617b640 10 monclient(hunting): authenticate will time out at 1541.585826s 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3e0effd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3e1013f650 0x7f3e10136740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:44885/0 (socket says 172.21.15.157:44885) 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3e0effd640 1 -- 172.21.15.157:0/137001353 learned_addr learned my addr 172.21.15.157:0/137001353 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3df7fff640 10 client.?.objecter ms_handle_connect 0x7f3e1013f650 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3df7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3df7fff640 10 client.?.objecter ms_handle_connect 0x7f3e100a9d70 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3df7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172757999 0 0) 0x7f3e10140bc0 con 0x7f3e100a9d70 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3df7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3df7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3df7fff640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.393+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dfc004eb0 con 0x7f3e100a9d70 2024-03-29T11:22:09.371 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.394+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294223944 0 0) 0x7f3e10140600 con 0x7f3e1013f650 2024-03-29T11:22:09.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.394+0000 7f3df7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.394+0000 7f3df7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.394+0000 7f3df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.394+0000 7f3df7fff640 10 monclient(hunting): my global_id is 4608 2024-03-29T11:22:09.372 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.394+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e10140bc0 con 0x7f3e1013f650 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.396+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1887475758 0 0) 0x7f3e10140bc0 con 0x7f3e1013f650 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.396+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e10140600 con 0x7f3e1013f650 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.396+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3e04001920 con 0x7f3e1013f650 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.396+0000 7f3df7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.396+0000 7f3df7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.396+0000 7f3df7fff640 10 monclient(hunting): dump: 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.373 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:09.374 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:09.374 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:09.374 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:09.374 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:09.374 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.396+0000 7f3df7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:09.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4249374633 0 0) 0x7f3e10140600 con 0x7f3e1013f650 2024-03-29T11:22:09.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 >> v1:172.21.15.157:6790/0 conn(0x7f3e100a9d70 legacy=0x7f3e1013da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 >> v1:172.21.15.195:6789/0 conn(0x7f3e101457f0 legacy=0x7f3e101eedc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:22:09.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:09.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 1 monclient: found mon.a 2024-03-29T11:22:09.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:09.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:22:09.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e10140fe0 con 0x7f3e1013f650 2024-03-29T11:22:09.377 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3df7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3e1617b640 5 monclient: authenticate success, global_id 4608 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3e1617b640 10 monclient: _renew_subs 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3e1617b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3e1617b640 1 -- 172.21.15.157:0/137001353 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1013e740 con 0x7f3e1013f650 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3e1617b640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3e1617b640 10 monclient: _renew_subs 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3e1617b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3e1617b640 1 -- 172.21.15.157:0/137001353 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e10142e50 con 0x7f3e1013f650 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.397+0000 7f3e1617b640 1 librados: init done 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.399+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e040035f0 con 0x7f3e1013f650 2024-03-29T11:22:09.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.399+0000 7f3df7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.399+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3e040039c0 con 0x7f3e1013f650 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.399+0000 7f3df7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.399+0000 7f3df7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.399+0000 7f3df7fff640 10 monclient: dump: 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:09.379 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:09.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.399+0000 7f3e0dffb640 4 set_mon_vals no callback set 2024-03-29T11:22:09.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.399+0000 7f3e1617b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:09.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.400+0000 7f3e1617b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:09.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.400+0000 7f3e1617b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:22:09.380 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.400+0000 7f3e1617b640 1 -- 172.21.15.157:0/137001353 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df0005080 con 0x7f3e1013f650 2024-03-29T11:22:09.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.401+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f3e04003b20 con 0x7f3e1013f650 2024-03-29T11:22:09.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.402+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 417662124 0 0) 0x7f3e040849d0 con 0x7f3e1013f650 2024-03-29T11:22:09.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.402+0000 7f3df7fff640 10 client.4608.objecter ms_dispatch 0x7f3e10000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:09.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.402+0000 7f3df7fff640 3 client.4608.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:09.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.402+0000 7f3df7fff640 3 client.4608.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:09.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.402+0000 7f3df7fff640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-29T11:22:09.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.403+0000 7f3e0f7fe640 10 monclient: get_auth_request con 0x7f3dfc06dd30 auth_method 0 2024-03-29T11:22:09.381 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.403+0000 7f3df7fff640 10 client.4608.objecter ms_handle_connect 0x7f3dfc06dd30 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.418+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e040502b0 con 0x7f3e1013f650 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.418+0000 7f3df7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.418+0000 7f3df7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:09.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.747+0000 7f3e1617b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:09.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.748+0000 7f3e1617b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:09.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.748+0000 7f3e1617b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:22:09.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.748+0000 7f3e1617b640 1 -- 172.21.15.157:0/137001353 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3df0005330 con 0x7f3e1013f650 2024-03-29T11:22:09.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.748+0000 7f3df7fff640 1 -- 172.21.15.157:0/137001353 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3435241529) 0x7f3e040521e0 con 0x7f3e1013f650 2024-03-29T11:22:09.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.748+0000 7f3df7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:09.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.748+0000 7f3df7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:09.727 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:22:09.727 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":11,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:21:38.840582+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:21:38.840582+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:09.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.754+0000 7f3e1617b640 10 librados: watch_flush enter 2024-03-29T11:22:09.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.754+0000 7f3e1617b640 10 librados: watch_flush exit 2024-03-29T11:22:09.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.754+0000 7f3e1617b640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-29T11:22:09.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.754+0000 7f3e1617b640 10 client.4608.objecter successfully canceled tick 2024-03-29T11:22:09.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.754+0000 7f3e1617b640 1 -- 172.21.15.157:0/137001353 >> v1:172.21.15.157:6816/1747058247 conn(0x7f3dfc06dd30 legacy=0x7f3dfc0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.754+0000 7f3e1617b640 10 monclient: shutdown 2024-03-29T11:22:09.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.754+0000 7f3e1617b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:09.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.754+0000 7f3e1617b640 1 -- 172.21.15.157:0/137001353 >> v1:172.21.15.157:6789/0 conn(0x7f3e1013f650 legacy=0x7f3e10136740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.755+0000 7f3e1617b640 1 -- 172.21.15.157:0/137001353 shutdown_connections 2024-03-29T11:22:09.732 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.755+0000 7f3e1617b640 1 -- 172.21.15.157:0/137001353 >> 172.21.15.157:0/137001353 conn(0x7f3e10066a00 msgr2=0x7f3e1013c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:09.733 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.756+0000 7f3e1617b640 1 -- 172.21.15.157:0/137001353 shutdown_connections 2024-03-29T11:22:09.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.758+0000 7f3e1617b640 1 -- 172.21.15.157:0/137001353 wait complete. 2024-03-29T11:22:09.735 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:09.758+0000 7f3e1617b640 1 librados: shutdown 2024-03-29T11:22:09.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:16.557 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:16.700 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.724+0000 7f3aa4afb640 10 monclient: get_monmap_and_config 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.724+0000 7f3aa4afb640 10 monclient: build_initial_monmap 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.724+0000 7f3aa4afb640 10 monclient: monmap: 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:22:16.725152+0000 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:22:16.725152+0000 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:22:16.701 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.725+0000 7f3aa4afb640 1 Processor -- start 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.725+0000 7f3aa4afb640 1 -- start start 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.725+0000 7f3aa4afb640 10 monclient: init 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.726+0000 7f3aa4afb640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.726+0000 7f3aa4afb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.726+0000 7f3aa4afb640 10 monclient(hunting): picked mon.noname-c con 0x7f3aa00b2790 addr v1:172.21.15.195:6789/0 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.726+0000 7f3aa4afb640 10 monclient(hunting): picked mon.noname-a con 0x7f3aa0143c80 addr v1:172.21.15.157:6789/0 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.726+0000 7f3aa4afb640 10 monclient(hunting): picked mon.noname-b con 0x7f3aa0146e30 addr v1:172.21.15.157:6790/0 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.726+0000 7f3aa4afb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa0146290 con 0x7f3aa0143c80 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.726+0000 7f3aa4afb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa00b3970 con 0x7f3aa00b2790 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.726+0000 7f3aa4afb640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa006ee80 con 0x7f3aa0146e30 2024-03-29T11:22:16.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.726+0000 7f3aa4afb640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:16.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.726+0000 7f3aa4afb640 10 monclient(hunting): authenticate will time out at 1548.923153s 2024-03-29T11:22:16.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.728+0000 7f3a9effd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3aa0143c80 0x7f3aa0146720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:46944/0 (socket says 172.21.15.157:46944) 2024-03-29T11:22:16.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.728+0000 7f3a9effd640 1 -- 172.21.15.157:0/1274906196 learned_addr learned my addr 172.21.15.157:0/1274906196 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:22:16.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.729+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390355522 0 0) 0x7f3aa0146290 con 0x7f3aa0143c80 2024-03-29T11:22:16.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.729+0000 7f3a9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:16.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.729+0000 7f3a9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:16.706 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.729+0000 7f3a9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:16.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.729+0000 7f3a9e7fc640 10 monclient(hunting): my global_id is 4611 2024-03-29T11:22:16.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.729+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a88003520 con 0x7f3aa0143c80 2024-03-29T11:22:16.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581063552 0 0) 0x7f3aa00b3970 con 0x7f3aa00b2790 2024-03-29T11:22:16.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:16.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:16.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:16.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient(hunting): my global_id is 4600 2024-03-29T11:22:16.707 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa0146290 con 0x7f3aa00b2790 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1338104024 0 0) 0x7f3a88003520 con 0x7f3aa0143c80 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 >> v1:172.21.15.157:6790/0 conn(0x7f3aa0146e30 legacy=0x7f3aa0151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 >> v1:172.21.15.195:6789/0 conn(0x7f3aa00b2790 legacy=0x7f3aa0143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.157:6789/0 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa006f160 con 0x7f3aa0143c80 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3a98001a10 con 0x7f3aa0143c80 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.708 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:16.709 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.730+0000 7f3a9e7fc640 10 monclient: dump: 2024-03-29T11:22:16.709 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:16.709 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:16.709 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:16.709 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:16.709 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.709 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:16.709 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.731+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 <== mon.0 v1:172.21.15.157:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a98001c30 con 0x7f3aa0143c80 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.731+0000 7f3a9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.731+0000 7f3a9e7fc640 1 -- 172.21.15.157:0/1274906196 <== mon.0 v1:172.21.15.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3a98001fc0 con 0x7f3aa0143c80 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.731+0000 7f3a9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.731+0000 7f3a9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.731+0000 7f3a9e7fc640 10 monclient: dump: 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:16.710 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:16.711 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.711 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:16.711 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:16.711 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:16.711 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:16.711 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:16.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.731+0000 7f3aa4afb640 5 monclient: authenticate success, global_id 4611 2024-03-29T11:22:16.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.731+0000 7f3aa4afb640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:16.711 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.731+0000 7f3aa4afb640 4 set_mon_vals no callback set 2024-03-29T11:22:16.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.733+0000 7f3aa4afb640 10 monclient: shutdown 2024-03-29T11:22:16.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.733+0000 7f3aa4afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:16.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.733+0000 7f3aa4afb640 1 -- 172.21.15.157:0/1274906196 >> v1:172.21.15.157:6789/0 conn(0x7f3aa0143c80 legacy=0x7f3aa0146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.733+0000 7f3aa4afb640 1 -- 172.21.15.157:0/1274906196 shutdown_connections 2024-03-29T11:22:16.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.733+0000 7f3aa4afb640 1 -- 172.21.15.157:0/1274906196 >> 172.21.15.157:0/1274906196 conn(0x7f3aa006d070 msgr2=0x7f3aa0066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:16.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.733+0000 7f3aa4afb640 1 -- 172.21.15.157:0/1274906196 shutdown_connections 2024-03-29T11:22:16.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.733+0000 7f3aa4afb640 1 -- 172.21.15.157:0/1274906196 wait complete. 2024-03-29T11:22:16.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.734+0000 7f3aa4afb640 10 monclient: build_initial_monmap 2024-03-29T11:22:16.713 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.734+0000 7f3aa4afb640 10 monclient: monmap: 2024-03-29T11:22:16.713 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.734+0000 7f3aa4afb640 1 librados: starting msgr at 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.734+0000 7f3aa4afb640 1 librados: starting objecter 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.734+0000 7f3aa4afb640 1 Processor -- start 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.735+0000 7f3aa4afb640 1 -- start start 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.735+0000 7f3aa4afb640 1 librados: setting wanted keys 2024-03-29T11:22:16.714 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.735+0000 7f3aa4afb640 1 librados: calling monclient init 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.735+0000 7f3aa4afb640 10 monclient: init 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.736+0000 7f3aa4afb640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.736+0000 7f3aa4afb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.736+0000 7f3aa4afb640 10 monclient(hunting): picked mon.noname-c con 0x7f3aa00b2790 addr v1:172.21.15.157:6790/0 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.736+0000 7f3aa4afb640 10 monclient(hunting): picked mon.noname-b con 0x7f3aa0143c80 addr v1:172.21.15.195:6789/0 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.736+0000 7f3aa4afb640 10 monclient(hunting): picked mon.noname-a con 0x7f3aa0146e30 addr v1:172.21.15.157:6789/0 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.736+0000 7f3aa4afb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa01dc420 con 0x7f3aa0146e30 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.736+0000 7f3aa4afb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa00b6580 con 0x7f3aa0143c80 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.736+0000 7f3aa4afb640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa0142f50 con 0x7f3aa00b2790 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.736+0000 7f3aa4afb640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.736+0000 7f3aa4afb640 10 monclient(hunting): authenticate will time out at 1548.933189s 2024-03-29T11:22:16.715 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.737+0000 7f3a9ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3aa0146e30 0x7f3aa01e6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:48368/0 (socket says 172.21.15.157:48368) 2024-03-29T11:22:16.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.737+0000 7f3a9ffff640 1 -- 172.21.15.157:0/4159768328 learned_addr learned my addr 172.21.15.157:0/4159768328 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:22:16.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.740+0000 7f3a87fff640 10 client.?.objecter ms_handle_connect 0x7f3aa0146e30 2024-03-29T11:22:16.716 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.740+0000 7f3a87fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.742+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772584304 0 0) 0x7f3aa01dc420 con 0x7f3aa0146e30 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.742+0000 7f3a87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.742+0000 7f3a87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.742+0000 7f3a87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.742+0000 7f3a87fff640 10 monclient(hunting): my global_id is 4614 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.742+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a8c004eb0 con 0x7f3aa0146e30 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 client.?.objecter ms_handle_connect 0x7f3aa00b2790 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139954604 0 0) 0x7f3aa0142f50 con 0x7f3aa00b2790 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:16.724 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa01dc420 con 0x7f3aa00b2790 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 client.?.objecter ms_handle_connect 0x7f3aa0143c80 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 430673993 0 0) 0x7f3a8c004eb0 con 0x7f3aa0146e30 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3aa0142f50 con 0x7f3aa0146e30 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3a94001750 con 0x7f3aa0146e30 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): dump: 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:16.725 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2239645701 0 0) 0x7f3aa01dc420 con 0x7f3aa00b2790 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a8c004eb0 con 0x7f3aa00b2790 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3a90001970 con 0x7f3aa00b2790 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.743+0000 7f3a87fff640 10 monclient(hunting): dump: 2024-03-29T11:22:16.726 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1274756840 0 0) 0x7f3a8c004eb0 con 0x7f3aa00b2790 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 >> v1:172.21.15.195:6789/0 conn(0x7f3aa0143c80 legacy=0x7f3aa01d8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 >> v1:172.21.15.157:6789/0 conn(0x7f3aa0146e30 legacy=0x7f3aa01e6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 1 monclient: found mon.c 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:16.727 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa0142030 con 0x7f3aa00b2790 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3aa4afb640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3aa4afb640 10 monclient: _renew_subs 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3aa4afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3aa4afb640 1 -- 172.21.15.157:0/4159768328 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa0142310 con 0x7f3aa00b2790 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3aa4afb640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3aa4afb640 10 monclient: _renew_subs 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3aa4afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3aa4afb640 1 -- 172.21.15.157:0/4159768328 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa01d9020 con 0x7f3aa00b2790 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3aa4afb640 1 librados: init done 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a90001ad0 con 0x7f3aa00b2790 2024-03-29T11:22:16.728 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f3a900038c0 con 0x7f3aa00b2790 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a87fff640 10 monclient: dump: 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:16.729 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:16.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.744+0000 7f3a9dffb640 4 set_mon_vals no callback set 2024-03-29T11:22:16.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.745+0000 7f3aa4afb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:16.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.746+0000 7f3aa4afb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:16.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.746+0000 7f3aa4afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:16.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.746+0000 7f3aa4afb640 1 -- 172.21.15.157:0/4159768328 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a80005080 con 0x7f3aa00b2790 2024-03-29T11:22:16.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.747+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f3a90004ad0 con 0x7f3aa00b2790 2024-03-29T11:22:16.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.748+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 417662124 0 0) 0x7f3a90083930 con 0x7f3aa00b2790 2024-03-29T11:22:16.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.748+0000 7f3a87fff640 10 client.4628.objecter ms_dispatch 0x7f3aa0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:16.730 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.748+0000 7f3a87fff640 3 client.4628.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.748+0000 7f3a87fff640 3 client.4628.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.748+0000 7f3a87fff640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.748+0000 7f3a9effd640 10 monclient: get_auth_request con 0x7f3a8c06dec0 auth_method 0 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.749+0000 7f3a87fff640 10 client.4628.objecter ms_handle_connect 0x7f3a8c06dec0 2024-03-29T11:22:16.734 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.757+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a9004f040 con 0x7f3aa00b2790 2024-03-29T11:22:16.734 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.757+0000 7f3a87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:16.734 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:16.757+0000 7f3a87fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:17.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.084+0000 7f3aa4afb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:17.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.084+0000 7f3aa4afb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:17.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.084+0000 7f3aa4afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:17.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.084+0000 7f3aa4afb640 1 -- 172.21.15.157:0/4159768328 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a80005330 con 0x7f3aa00b2790 2024-03-29T11:22:17.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.088+0000 7f3a87fff640 1 -- 172.21.15.157:0/4159768328 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3435241529) 0x7f3a90051140 con 0x7f3aa00b2790 2024-03-29T11:22:17.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.088+0000 7f3a87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:17.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.088+0000 7f3a87fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:17.065 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:22:17.065 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":11,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:21:38.840582+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:21:38.840582+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:17.068 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.091+0000 7f3aa4afb640 10 librados: watch_flush enter 2024-03-29T11:22:17.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.092+0000 7f3aa4afb640 10 librados: watch_flush exit 2024-03-29T11:22:17.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.092+0000 7f3aa4afb640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:22:17.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.092+0000 7f3aa4afb640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:22:17.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.092+0000 7f3aa4afb640 1 -- 172.21.15.157:0/4159768328 >> v1:172.21.15.157:6816/1747058247 conn(0x7f3a8c06dec0 legacy=0x7f3a8c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.092+0000 7f3aa4afb640 10 monclient: shutdown 2024-03-29T11:22:17.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.092+0000 7f3aa4afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:17.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.092+0000 7f3aa4afb640 1 -- 172.21.15.157:0/4159768328 >> v1:172.21.15.157:6790/0 conn(0x7f3aa00b2790 legacy=0x7f3aa01db500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.094+0000 7f3aa4afb640 1 -- 172.21.15.157:0/4159768328 shutdown_connections 2024-03-29T11:22:17.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.094+0000 7f3aa4afb640 1 -- 172.21.15.157:0/4159768328 >> 172.21.15.157:0/4159768328 conn(0x7f3aa006d070 msgr2=0x7f3aa013d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:17.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.094+0000 7f3aa4afb640 1 -- 172.21.15.157:0/4159768328 shutdown_connections 2024-03-29T11:22:17.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.094+0000 7f3aa4afb640 1 -- 172.21.15.157:0/4159768328 wait complete. 2024-03-29T11:22:17.072 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:17.094+0000 7f3aa4afb640 1 librados: shutdown 2024-03-29T11:22:17.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:23.896 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:24.045 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.068+0000 7fd905e38640 10 monclient: get_monmap_and_config 2024-03-29T11:22:24.045 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.068+0000 7fd905e38640 10 monclient: build_initial_monmap 2024-03-29T11:22:24.045 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.068+0000 7fd905e38640 10 monclient: monmap: 2024-03-29T11:22:24.045 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:22:24.046 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:24.046 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:22:24.070033+0000 2024-03-29T11:22:24.046 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:22:24.070033+0000 2024-03-29T11:22:24.046 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:24.046 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:24.046 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:22:24.046 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:22:24.046 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:22:24.046 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:24.047 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.070+0000 7fd905e38640 1 Processor -- start 2024-03-29T11:22:24.048 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.070+0000 7fd905e38640 1 -- start start 2024-03-29T11:22:24.048 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.070+0000 7fd905e38640 10 monclient: init 2024-03-29T11:22:24.048 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd905e38640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:24.048 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd905e38640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:24.049 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd905e38640 10 monclient(hunting): picked mon.noname-b con 0x7fd8f80389e0 addr v1:172.21.15.157:6790/0 2024-03-29T11:22:24.049 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd905e38640 10 monclient(hunting): picked mon.noname-a con 0x7fd8f8038dc0 addr v1:172.21.15.157:6789/0 2024-03-29T11:22:24.049 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd905e38640 10 monclient(hunting): picked mon.noname-c con 0x7fd8f8030660 addr v1:172.21.15.195:6789/0 2024-03-29T11:22:24.049 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd905e38640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80c54c0 con 0x7fd8f8038dc0 2024-03-29T11:22:24.049 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd905e38640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f8036430 con 0x7fd8f8030660 2024-03-29T11:22:24.049 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd905e38640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f8039840 con 0x7fd8f80389e0 2024-03-29T11:22:24.049 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd905e38640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:24.049 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd905e38640 10 monclient(hunting): authenticate will time out at 1556.268031s 2024-03-29T11:22:24.049 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd904e36640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fd8f80389e0 0x7fd8f8033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:41122/0 (socket says 172.21.15.157:41122) 2024-03-29T11:22:24.049 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.071+0000 7fd904e36640 1 -- 172.21.15.157:0/3107476538 learned_addr learned my addr 172.21.15.157:0/3107476538 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:22:24.050 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.072+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268528836 0 0) 0x7fd8f8039840 con 0x7fd8f80389e0 2024-03-29T11:22:24.050 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.072+0000 7fd8ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:24.050 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.072+0000 7fd8ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:24.050 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.072+0000 7fd8ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:24.050 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.072+0000 7fd8ff7fe640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:22:24.050 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.072+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f0003520 con 0x7fd8f80389e0 2024-03-29T11:22:24.052 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.075+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024640800 0 0) 0x7fd8f8036430 con 0x7fd8f8030660 2024-03-29T11:22:24.052 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.075+0000 7fd8ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:24.052 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.075+0000 7fd8ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:24.052 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.075+0000 7fd8ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:24.052 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.075+0000 7fd8ff7fe640 10 monclient(hunting): my global_id is 4606 2024-03-29T11:22:24.052 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.075+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f8039840 con 0x7fd8f8030660 2024-03-29T11:22:24.053 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.075+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1074475447 0 0) 0x7fd8f8039840 con 0x7fd8f8030660 2024-03-29T11:22:24.053 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.075+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 >> v1:172.21.15.157:6790/0 conn(0x7fd8f80389e0 legacy=0x7fd8f8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.053 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.075+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 >> v1:172.21.15.157:6789/0 conn(0x7fd8f8038dc0 legacy=0x7fd8f80cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 1 monclient: found mon.noname-c 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.195:6789/0 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f8035690 con 0x7fd8f8030660 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd90004f730 con 0x7fd8f8030660 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:24.055 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: dump: 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd905e38640 5 monclient: authenticate success, global_id 4606 2024-03-29T11:22:24.056 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd905e38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 <== mon.1 v1:172.21.15.195:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd900075f10 con 0x7fd8f8030660 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 1 -- 172.21.15.157:0/3107476538 <== mon.1 v1:172.21.15.195:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd9000758b0 con 0x7fd8f8030660 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd8ff7fe640 10 monclient: dump: 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:24.057 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:24.058 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:24.058 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd905e38640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:24.058 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.078+0000 7fd905e38640 4 set_mon_vals no callback set 2024-03-29T11:22:24.060 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.079+0000 7fd905e38640 10 monclient: shutdown 2024-03-29T11:22:24.060 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.079+0000 7fd905e38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:24.060 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.079+0000 7fd905e38640 1 -- 172.21.15.157:0/3107476538 >> v1:172.21.15.195:6789/0 conn(0x7fd8f8030660 legacy=0x7fd8f80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.060 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.079+0000 7fd905e38640 1 -- 172.21.15.157:0/3107476538 shutdown_connections 2024-03-29T11:22:24.060 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.079+0000 7fd905e38640 1 -- 172.21.15.157:0/3107476538 >> 172.21.15.157:0/3107476538 conn(0x7fd8f8005c00 msgr2=0x7fd8f8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:24.060 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.079+0000 7fd905e38640 1 -- 172.21.15.157:0/3107476538 shutdown_connections 2024-03-29T11:22:24.060 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.079+0000 7fd905e38640 1 -- 172.21.15.157:0/3107476538 wait complete. 2024-03-29T11:22:24.060 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.080+0000 7fd905e38640 10 monclient: build_initial_monmap 2024-03-29T11:22:24.060 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.080+0000 7fd905e38640 10 monclient: monmap: 2024-03-29T11:22:24.060 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.081+0000 7fd905e38640 1 librados: starting msgr at 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.081+0000 7fd905e38640 1 librados: starting objecter 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.081+0000 7fd905e38640 1 Processor -- start 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.081+0000 7fd905e38640 1 -- start start 2024-03-29T11:22:24.061 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.081+0000 7fd905e38640 1 librados: setting wanted keys 2024-03-29T11:22:24.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.081+0000 7fd905e38640 1 librados: calling monclient init 2024-03-29T11:22:24.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.081+0000 7fd905e38640 10 monclient: init 2024-03-29T11:22:24.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.082+0000 7fd905e38640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:24.062 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.082+0000 7fd905e38640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:24.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.082+0000 7fd904e36640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7fd8f8030660 0x7fd8f802f860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:41426/0 (socket says 172.21.15.157:41426) 2024-03-29T11:22:24.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.082+0000 7fd904e36640 1 -- 172.21.15.157:0/3097058933 learned_addr learned my addr 172.21.15.157:0/3097058933 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:22:24.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.082+0000 7fd905e38640 10 monclient(hunting): picked mon.noname-c con 0x7fd8f8030660 addr v1:172.21.15.157:6790/0 2024-03-29T11:22:24.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.082+0000 7fd8fd7fa640 10 client.?.objecter ms_handle_connect 0x7fd8f8030660 2024-03-29T11:22:24.063 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.082+0000 7fd8fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:24.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.085+0000 7fd905e38640 10 monclient(hunting): picked mon.noname-a con 0x7fd8f80389e0 addr v1:172.21.15.157:6789/0 2024-03-29T11:22:24.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.086+0000 7fd905e38640 10 monclient(hunting): picked mon.noname-b con 0x7fd8f802ff70 addr v1:172.21.15.195:6789/0 2024-03-29T11:22:24.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.086+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80c0a80 con 0x7fd8f80389e0 2024-03-29T11:22:24.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.086+0000 7fd8fd7fa640 10 client.?.objecter ms_handle_connect 0x7fd8f80389e0 2024-03-29T11:22:24.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.086+0000 7fd8fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:24.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.086+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f802f360 con 0x7fd8f802ff70 2024-03-29T11:22:24.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.086+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80c5650 con 0x7fd8f8030660 2024-03-29T11:22:24.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.087+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 <== mon.0 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251890782 0 0) 0x7fd8f80c0a80 con 0x7fd8f80389e0 2024-03-29T11:22:24.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.087+0000 7fd905e38640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:24.064 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.087+0000 7fd905e38640 10 monclient(hunting): authenticate will time out at 1556.284193s 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.087+0000 7fd8fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.087+0000 7fd8fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.087+0000 7fd8fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.087+0000 7fd8fd7fa640 10 monclient(hunting): my global_id is 4617 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.087+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e4004eb0 con 0x7fd8f80389e0 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.088+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 <== mon.0 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819546818 0 0) 0x7fd8e4004eb0 con 0x7fd8f80389e0 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.088+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 --> v1:172.21.15.157:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8f80c0a80 con 0x7fd8f80389e0 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.088+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 <== mon.0 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd8ec0017d0 con 0x7fd8f80389e0 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.088+0000 7fd8fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.088+0000 7fd8fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:24.065 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.088+0000 7fd8fd7fa640 10 monclient(hunting): dump: 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.088+0000 7fd8fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.088+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 <== mon.0 v1:172.21.15.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2291875785 0 0) 0x7fd8f80c0a80 con 0x7fd8f80389e0 2024-03-29T11:22:24.066 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.088+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 >> v1:172.21.15.157:6790/0 conn(0x7fd8f8030660 legacy=0x7fd8f802f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 >> v1:172.21.15.195:6789/0 conn(0x7fd8f802ff70 legacy=0x7fd8f80c0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 1 monclient: found mon.a 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f80c57e0 con 0x7fd8f80389e0 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd905e38640 5 monclient: authenticate success, global_id 4617 2024-03-29T11:22:24.069 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd905e38640 10 monclient: _renew_subs 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd905e38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 --> v1:172.21.15.157:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f80c8420 con 0x7fd8f80389e0 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd905e38640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 <== mon.0 v1:172.21.15.157:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8ec001930 con 0x7fd8f80389e0 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.089+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 <== mon.0 v1:172.21.15.157:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7fd8ec003810 con 0x7fd8f80389e0 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.090+0000 7fd8fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.090+0000 7fd8fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.090+0000 7fd8fd7fa640 10 monclient: dump: 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:24.070 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.090+0000 7fd905e38640 10 monclient: _renew_subs 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.090+0000 7fd905e38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.090+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 --> v1:172.21.15.157:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f802d5a0 con 0x7fd8f80389e0 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.091+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 <== mon.0 v1:172.21.15.157:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7fd8ec0197e0 con 0x7fd8f80389e0 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.092+0000 7fd905e38640 1 librados: init done 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.092+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 <== mon.0 v1:172.21.15.157:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 417662124 0 0) 0x7fd8ec083810 con 0x7fd8f80389e0 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.092+0000 7fd8fd7fa640 10 client.4617.objecter ms_dispatch 0x7fd900000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.092+0000 7fd8fd7fa640 3 client.4617.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:24.071 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.092+0000 7fd8fd7fa640 3 client.4617.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:24.072 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.092+0000 7fd8fe7fc640 4 set_mon_vals no callback set 2024-03-29T11:22:24.072 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.093+0000 7fd8fd7fa640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.094+0000 7fd904e36640 10 monclient: get_auth_request con 0x7fd8e406db30 auth_method 0 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.094+0000 7fd8fd7fa640 10 client.4617.objecter ms_handle_connect 0x7fd8e406db30 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.095+0000 7fd905e38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.095+0000 7fd905e38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.095+0000 7fd905e38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.095+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e0005080 con 0x7fd8f80389e0 2024-03-29T11:22:24.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.103+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 <== mon.0 v1:172.21.15.157:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8ec04ef80 con 0x7fd8f80389e0 2024-03-29T11:22:24.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.103+0000 7fd8fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:24.080 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.103+0000 7fd8fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:24.409 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.432+0000 7fd905e38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:24.409 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.432+0000 7fd905e38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:24.409 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.432+0000 7fd905e38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.157:6789/0 2024-03-29T11:22:24.409 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.432+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 --> v1:172.21.15.157:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8e0005330 con 0x7fd8f80389e0 2024-03-29T11:22:24.410 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.434+0000 7fd8fd7fa640 1 -- 172.21.15.157:0/3097058933 <== mon.0 v1:172.21.15.157:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3435241529) 0x7fd8ec082ec0 con 0x7fd8f80389e0 2024-03-29T11:22:24.411 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.434+0000 7fd8fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:24.411 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.434+0000 7fd8fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:24.413 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:22:24.413 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":11,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:21:38.840582+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:21:38.840582+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:24.415 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.439+0000 7fd905e38640 10 librados: watch_flush enter 2024-03-29T11:22:24.417 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.440+0000 7fd905e38640 10 librados: watch_flush exit 2024-03-29T11:22:24.417 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.440+0000 7fd905e38640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-29T11:22:24.417 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.440+0000 7fd905e38640 10 client.4617.objecter successfully canceled tick 2024-03-29T11:22:24.417 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.440+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 >> v1:172.21.15.157:6816/1747058247 conn(0x7fd8e406db30 legacy=0x7fd8e406fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.417 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.440+0000 7fd905e38640 10 monclient: shutdown 2024-03-29T11:22:24.418 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.440+0000 7fd905e38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:24.418 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.440+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 >> v1:172.21.15.157:6789/0 conn(0x7fd8f80389e0 legacy=0x7fd8f80bfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.418 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.440+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 shutdown_connections 2024-03-29T11:22:24.418 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.440+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 >> 172.21.15.157:0/3097058933 conn(0x7fd8f8005c00 msgr2=0x7fd8f8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:24.418 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.441+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 shutdown_connections 2024-03-29T11:22:24.418 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.441+0000 7fd905e38640 1 -- 172.21.15.157:0/3097058933 wait complete. 2024-03-29T11:22:24.418 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:24.441+0000 7fd905e38640 1 librados: shutdown 2024-03-29T11:22:24.433 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:31.239 DEBUG:teuthology.orchestra.run.smithi157:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:31.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.401+0000 7f6b761cb640 10 monclient: get_monmap_and_config 2024-03-29T11:22:31.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.401+0000 7f6b761cb640 10 monclient: build_initial_monmap 2024-03-29T11:22:31.378 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.401+0000 7f6b761cb640 10 monclient: monmap: 2024-03-29T11:22:31.378 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:22:31.378 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:31.378 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:22:31.402818+0000 2024-03-29T11:22:31.378 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:22:31.402818+0000 2024-03-29T11:22:31.378 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:31.378 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:31.379 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:22:31.379 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c 2024-03-29T11:22:31.379 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-b 2024-03-29T11:22:31.379 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:31.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.403+0000 7f6b761cb640 1 Processor -- start 2024-03-29T11:22:31.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.403+0000 7f6b761cb640 1 -- start start 2024-03-29T11:22:31.379 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.403+0000 7f6b761cb640 10 monclient: init 2024-03-29T11:22:31.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b761cb640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:31.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b761cb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:31.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b761cb640 10 monclient(hunting): picked mon.noname-c con 0x7f6b701457f0 addr v1:172.21.15.195:6789/0 2024-03-29T11:22:31.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b761cb640 10 monclient(hunting): picked mon.noname-b con 0x7f6b7013f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:22:31.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b761cb640 10 monclient(hunting): picked mon.noname-a con 0x7f6b700a9d70 addr v1:172.21.15.157:6789/0 2024-03-29T11:22:31.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b761cb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b700aaef0 con 0x7f6b700a9d70 2024-03-29T11:22:31.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b761cb640 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b70145bd0 con 0x7f6b701457f0 2024-03-29T11:22:31.382 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b761cb640 1 -- --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b70145eb0 con 0x7f6b7013f650 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b761cb640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b761cb640 10 monclient(hunting): authenticate will time out at 1563.601150s 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b6f7fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6b701457f0 0x7f6b7013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:57594/0 (socket says 172.21.15.157:57594) 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b67fff640 1 --1- >> v1:172.21.15.157:6790/0 conn(0x7f6b7013f650 0x7f6b700b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6790/0 says I am v1:172.21.15.157:42876/0 (socket says 172.21.15.157:42876) 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b6f7fe640 1 -- 172.21.15.157:0/4073398643 learned_addr learned my addr 172.21.15.157:0/4073398643 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b6effd640 1 -- 172.21.15.157:0/4073398643 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497985971 0 0) 0x7f6b70145eb0 con 0x7f6b7013f650 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b6effd640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.404+0000 7f6b6effd640 1 -- 172.21.15.157:0/4073398643 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b50003520 con 0x7f6b7013f650 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.405+0000 7f6b6effd640 1 -- 172.21.15.157:0/4073398643 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990561946 0 0) 0x7f6b70145bd0 con 0x7f6b701457f0 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.405+0000 7f6b6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.405+0000 7f6b6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.383 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.405+0000 7f6b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.405+0000 7f6b6effd640 10 monclient(hunting): my global_id is 4609 2024-03-29T11:22:31.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.405+0000 7f6b6effd640 1 -- 172.21.15.157:0/4073398643 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b70145eb0 con 0x7f6b701457f0 2024-03-29T11:22:31.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.405+0000 7f6b6effd640 1 -- 172.21.15.157:0/4073398643 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2244616236 0 0) 0x7f6b50003520 con 0x7f6b7013f650 2024-03-29T11:22:31.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.405+0000 7f6b6effd640 1 -- 172.21.15.157:0/4073398643 >> v1:172.21.15.195:6789/0 conn(0x7f6b701457f0 legacy=0x7f6b7013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.384 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.405+0000 7f6b6effd640 1 -- 172.21.15.157:0/4073398643 >> v1:172.21.15.157:6789/0 conn(0x7f6b700a9d70 legacy=0x7f6b7014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:31.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:31.386 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 1 monclient: found mon.noname-b 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.157:6790/0 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 1 -- 172.21.15.157:0/4073398643 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b70146340 con 0x7f6b7013f650 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 10 monclient: _finish_auth 0 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 1 -- 172.21.15.157:0/4073398643 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6b580015c0 con 0x7f6b7013f650 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b6effd640 10 monclient: dump: 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:31.387 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b761cb640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.406+0000 7f6b761cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.407+0000 7f6b6effd640 1 -- 172.21.15.157:0/4073398643 <== mon.2 v1:172.21.15.157:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b58001720 con 0x7f6b7013f650 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.407+0000 7f6b6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.407+0000 7f6b761cb640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.407+0000 7f6b761cb640 4 set_mon_vals no callback set 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.408+0000 7f6b761cb640 10 monclient: shutdown 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.408+0000 7f6b761cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.408+0000 7f6b761cb640 1 -- 172.21.15.157:0/4073398643 >> v1:172.21.15.157:6790/0 conn(0x7f6b7013f650 legacy=0x7f6b700b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.408+0000 7f6b761cb640 1 -- 172.21.15.157:0/4073398643 shutdown_connections 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.408+0000 7f6b761cb640 1 -- 172.21.15.157:0/4073398643 >> 172.21.15.157:0/4073398643 conn(0x7f6b70066a00 msgr2=0x7f6b70066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:31.388 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.408+0000 7f6b761cb640 1 -- 172.21.15.157:0/4073398643 shutdown_connections 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.408+0000 7f6b761cb640 1 -- 172.21.15.157:0/4073398643 wait complete. 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.409+0000 7f6b761cb640 10 monclient: build_initial_monmap 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.409+0000 7f6b761cb640 10 monclient: monmap: 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:epoch 0 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:last_changed 0.000000 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:created 0.000000 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.noname-a 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.noname-b 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.noname-c 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.409+0000 7f6b761cb640 1 librados: starting msgr at 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.409+0000 7f6b761cb640 1 librados: starting objecter 2024-03-29T11:22:31.389 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.409+0000 7f6b761cb640 1 Processor -- start 2024-03-29T11:22:31.390 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.410+0000 7f6b761cb640 1 -- start start 2024-03-29T11:22:31.390 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.410+0000 7f6b761cb640 1 librados: setting wanted keys 2024-03-29T11:22:31.390 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.410+0000 7f6b761cb640 1 librados: calling monclient init 2024-03-29T11:22:31.390 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.410+0000 7f6b761cb640 10 monclient: init 2024-03-29T11:22:31.395 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b761cb640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:31.395 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b761cb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b761cb640 10 monclient(hunting): picked mon.noname-b con 0x7f6b700a9d70 addr v1:172.21.15.195:6789/0 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b761cb640 10 monclient(hunting): picked mon.noname-c con 0x7f6b7013f650 addr v1:172.21.15.157:6790/0 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b761cb640 10 monclient(hunting): picked mon.noname-a con 0x7f6b701457f0 addr v1:172.21.15.157:6789/0 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b6f7fe640 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f6b700a9d70 0x7f6b70139cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.157:58106/0 (socket says 172.21.15.157:58106) 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b6f7fe640 1 -- 172.21.15.157:0/1634594943 learned_addr learned my addr 172.21.15.157:0/1634594943 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b761cb640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b70071c40 con 0x7f6b701457f0 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b6cff9640 10 client.?.objecter ms_handle_connect 0x7f6b7013f650 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.411+0000 7f6b761cb640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b70070930 con 0x7f6b700a9d70 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b761cb640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.157:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b70070380 con 0x7f6b7013f650 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b761cb640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 client.?.objecter ms_handle_connect 0x7f6b700a9d70 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b761cb640 10 monclient(hunting): authenticate will time out at 1563.609090s 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.2 v1:172.21.15.157:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106881652 0 0) 0x7f6b70070380 con 0x7f6b7013f650 2024-03-29T11:22:31.396 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.157:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b54004eb0 con 0x7f6b7013f650 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015682553 0 0) 0x7f6b70070930 con 0x7f6b700a9d70 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): my global_id is 4612 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b70070380 con 0x7f6b700a9d70 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.2 v1:172.21.15.157:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3804151113 0 0) 0x7f6b54004eb0 con 0x7f6b7013f650 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.157:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b70070930 con 0x7f6b7013f650 2024-03-29T11:22:31.397 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.2 v1:172.21.15.157:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6b58004980 con 0x7f6b7013f650 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): dump: 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:31.398 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.2 v1:172.21.15.157:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1795942532 0 0) 0x7f6b70070930 con 0x7f6b7013f650 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 >> v1:172.21.15.195:6789/0 conn(0x7f6b700a9d70 legacy=0x7f6b70139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.412+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 >> v1:172.21.15.157:6789/0 conn(0x7f6b701457f0 legacy=0x7f6b701dbed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b6cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b6cff9640 1 monclient: found mon.c 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b6cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b6cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.157:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b70070510 con 0x7f6b7013f650 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b6cff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b6cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b761cb640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b761cb640 10 monclient: _renew_subs 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b761cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b761cb640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.157:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7006fd30 con 0x7f6b7013f650 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b761cb640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b761cb640 10 monclient: _renew_subs 2024-03-29T11:22:31.399 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b761cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b761cb640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.157:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b701423d0 con 0x7f6b7013f650 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.413+0000 7f6b761cb640 1 librados: init done 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.415+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.2 v1:172.21.15.157:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b58001990 con 0x7f6b7013f650 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.415+0000 7f6b6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.415+0000 7f6b6e7fc640 4 set_mon_vals no callback set 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.415+0000 7f6b65ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.415+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.2 v1:172.21.15.157:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1463825438 0 0) 0x7f6b58004f30 con 0x7f6b7013f650 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.415+0000 7f6b6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.415+0000 7f6b6cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.415+0000 7f6b6cff9640 10 monclient: dump: 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:epoch 1 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:fsid 0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a 2024-03-29T11:22:31.400 INFO:teuthology.misc.health.smithi157.stderr:last_changed 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:created 2024-03-29T11:11:04.607260+0000 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:election_strategy: 1 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:0: v1:172.21.15.157:6789/0 mon.a 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:1: v1:172.21.15.195:6789/0 mon.b 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:2: v1:172.21.15.157:6790/0 mon.c 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr: 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.416+0000 7f6b65ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.416+0000 7f6b65ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.416+0000 7f6b65ffb640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b48005080 con 0x7f6b7013f650 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.417+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.2 v1:172.21.15.157:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2593521613 0 0) 0x7f6b5801af00 con 0x7f6b7013f650 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.418+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.2 v1:172.21.15.157:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 417662124 0 0) 0x7f6b58084c10 con 0x7f6b7013f650 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.418+0000 7f6b6cff9640 10 client.4640.objecter ms_dispatch 0x7f6b70000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.418+0000 7f6b6cff9640 3 client.4640.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:31.401 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.418+0000 7f6b6cff9640 3 client.4640.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:31.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.418+0000 7f6b6f7fe640 10 monclient: get_auth_request con 0x7f6b5406dc90 auth_method 0 2024-03-29T11:22:31.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.418+0000 7f6b6cff9640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:22:31.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.418+0000 7f6b6cff9640 10 client.4640.objecter ms_handle_connect 0x7f6b5406dc90 2024-03-29T11:22:31.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.425+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.2 v1:172.21.15.157:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b580504f0 con 0x7f6b7013f650 2024-03-29T11:22:31.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.425+0000 7f6b6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:31.402 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.425+0000 7f6b6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:31.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.724+0000 7f6b65ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:31.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.724+0000 7f6b65ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:31.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.724+0000 7f6b65ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.157:6790/0 2024-03-29T11:22:31.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.724+0000 7f6b65ffb640 1 -- 172.21.15.157:0/1634594943 --> v1:172.21.15.157:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b48005330 con 0x7f6b7013f650 2024-03-29T11:22:31.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.725+0000 7f6b6cff9640 1 -- 172.21.15.157:0/1634594943 <== mon.2 v1:172.21.15.157:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3435241529) 0x7f6b58052420 con 0x7f6b7013f650 2024-03-29T11:22:31.701 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.725+0000 7f6b6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:31.702 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.725+0000 7f6b6cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:31.702 INFO:teuthology.misc.health.smithi157.stdout: 2024-03-29T11:22:31.702 INFO:teuthology.misc.health.smithi157.stdout:{"epoch":11,"fsid":"0e3ccc46-2c6b-45a6-b72e-1cf573cecd2a","created":"2024-03-29T11:11:33.887691+0000","modified":"2024-03-29T11:21:38.840582+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:21:38.840582+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fafbdcd9-9c2e-4f7a-86d5-875cb351874c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"9cc8713a-d703-408a-9b9d-98b8e5aace07","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f55dd07-77ef-4e1b-9463-6e172452c432","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5315b994-aa27-42bb-84d2-00a227ac264e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ba4bdf82-db18-41a4-b021-a96362bfefb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"527f85da-b570-45f6-bd5f-81434c94f48a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6eeab36f-56c0-4923-bf31-ac9b0c845f6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"722f1ec5-66b1-4d7c-993d-25ec8b415f2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:31.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.727+0000 7f6b65ffb640 10 librados: watch_flush enter 2024-03-29T11:22:31.704 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.727+0000 7f6b65ffb640 10 librados: watch_flush exit 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.727+0000 7f6b65ffb640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.727+0000 7f6b65ffb640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.727+0000 7f6b65ffb640 1 -- 172.21.15.157:0/1634594943 >> v1:172.21.15.157:6816/1747058247 conn(0x7f6b5406dc90 legacy=0x7f6b54070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.727+0000 7f6b65ffb640 10 monclient: shutdown 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.727+0000 7f6b65ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.727+0000 7f6b65ffb640 1 -- 172.21.15.157:0/1634594943 >> v1:172.21.15.157:6790/0 conn(0x7f6b7013f650 legacy=0x7f6b701db7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.727+0000 7f6b65ffb640 1 -- 172.21.15.157:0/1634594943 shutdown_connections 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.727+0000 7f6b65ffb640 1 -- 172.21.15.157:0/1634594943 >> 172.21.15.157:0/1634594943 conn(0x7f6b70066a00 msgr2=0x7f6b7013ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.728+0000 7f6b65ffb640 1 -- 172.21.15.157:0/1634594943 shutdown_connections 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.728+0000 7f6b65ffb640 1 -- 172.21.15.157:0/1634594943 wait complete. 2024-03-29T11:22:31.705 INFO:teuthology.misc.health.smithi157.stderr:2024-03-29T11:22:31.728+0000 7f6b65ffb640 1 librados: shutdown 2024-03-29T11:22:31.718 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:31.718 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:22:31.720 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:22:31.720 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:22:31.720 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:31.752 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:22:31.752 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:22:31.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:31.824 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:22:31.824 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:22:31.824 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:31.895 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:22:31.896 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:22:31.896 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:31.965 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:22:31.966 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:22:31.966 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:32.004 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:22:32.004 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:22:32.004 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:32.078 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:22:32.078 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:22:32.078 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:32.156 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:22:32.156 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:22:32.156 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:32.226 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:22:32.226 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:22:32.226 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:22:32.226 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:32.250 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:22:32.251 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:22:32.251 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:32.290 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:22:32.291 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:22:32.291 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:22:32.291 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:32.303 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:22:32.303 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:22:32.303 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:32.354 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:22:32.355 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:22:32.355 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:22:32.365 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:22:32.365 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:22:32.365 DEBUG:teuthology.orchestra.run.smithi157:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:22:32.428 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T11:22:32.428 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:22:36.116 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T11:22:36.117 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:22:36.254 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T11:22:36.254 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:22:36.335 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T11:22:36.335 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:22:36.425 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T11:22:36.425 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:22:38.685 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T11:22:38.686 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:22:38.755 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T11:22:38.755 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:22:38.811 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T11:22:38.812 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:22:38.870 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:22:38.872 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676/data/mon.a.tgz 2024-03-29T11:22:38.872 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-29T11:22:38.891 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.241sHryVPm 2024-03-29T11:22:38.891 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.241sHryVPm 2024-03-29T11:22:39.016 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.241sHryVPm 2024-03-29T11:22:39.104 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.241sHryVPm is 211KB 2024-03-29T11:22:39.198 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.241sHryVPm 2024-03-29T11:22:39.213 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676/data/mon.c.tgz 2024-03-29T11:22:39.213 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-29T11:22:39.267 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.7dvYBsqSoF 2024-03-29T11:22:39.267 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.7dvYBsqSoF 2024-03-29T11:22:39.368 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.7dvYBsqSoF 2024-03-29T11:22:39.453 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.7dvYBsqSoF is 212KB 2024-03-29T11:22:39.548 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.7dvYBsqSoF 2024-03-29T11:22:39.563 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676/data/mon.b.tgz 2024-03-29T11:22:39.563 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-03-29T11:22:39.581 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.NDVqitbZlQ 2024-03-29T11:22:39.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.NDVqitbZlQ 2024-03-29T11:22:39.722 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.NDVqitbZlQ 2024-03-29T11:22:39.811 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.NDVqitbZlQ is 212KB 2024-03-29T11:22:39.876 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.NDVqitbZlQ 2024-03-29T11:22:39.892 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:22:39.892 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:22:39.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:22:39.974 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:22:40.015 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:22:40.015 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:22:40.017 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676/remote/smithi157/crash 2024-03-29T11:22:40.018 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:22:40.045 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676/remote/smithi195/crash 2024-03-29T11:22:40.045 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:22:40.082 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:22:40.082 DEBUG:teuthology.orchestra.run.smithi157:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:22:40.089 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:22:40.151 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39220.log 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40550.log 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41723.log 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35032.log 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39220.log.gz 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41723.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40550.log.gz 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.35032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38399.log 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41723.log.gz 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41293.log 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35032.log.gz 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41762.log 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39103.log 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38399.log.gz 2024-03-29T11:22:40.152 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39298.log 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41293.log.gz 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38321.log 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39103.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41762.log.gz 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41410.log 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39103.log.gz 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38713.log 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39298.log.gz 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38674.log 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38321.log.gz 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40394.log 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41410.log.gz 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr: 86.5%/var/log/ceph/ceph-client.admin.38674.log: -- replaced with /var/log/ceph/ceph-client.admin.38713.log.gz 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41254.log 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40745.log 2024-03-29T11:22:40.153 INFO:teuthology.orchestra.run.smithi157.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38674.log.gz 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39337.log 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40394.log.gz 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40589.log 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40745.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41254.log.gz 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41371.log 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40589.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40745.log.gz 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38947.log 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39337.log.gz 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40589.log.gz 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41684.log 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41371.log: /var/log/ceph/ceph-client.admin.38947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41567.log 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40628.log 2024-03-29T11:22:40.154 INFO:teuthology.orchestra.run.smithi157.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41371.log.gz 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38947.log.gz 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40862.log 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41684.log.gz 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39181.log 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41567.log.gz 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40862.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40628.log.gzgzip 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39064.log 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38752.log 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40862.log.gz 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39883.log 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39181.log.gz 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38438.log 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39064.log.gz 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38516.log 2024-03-29T11:22:40.155 INFO:teuthology.orchestra.run.smithi157.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38752.log.gz 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39883.log.gz 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38516.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38438.log.gz 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40199.log 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38516.log.gz 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38477.log 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41332.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28831.log 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40199.log.gz 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38869.log 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gz 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.tmp-client.admin.28831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40511.log 2024-03-29T11:22:40.156 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28831.log.gz 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38477.log.gz 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38869.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40511.log: gzip -5 --verbose 86.2% -- /var/log/ceph/ceph-client.admin.40706.log -- replaced with /var/log/ceph/ceph-client.admin.38869.log.gz 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34638.log 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40511.log.gz 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40316.log 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr: 93.6%/var/log/ceph/ceph-client.admin.34638.log: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39688.log 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40706.log.gz 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38908.log 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34638.log.gz 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40667.log 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40316.log.gz 2024-03-29T11:22:40.157 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40001.log 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39688.log.gz 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40667.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38908.log.gz 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39805.log 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40667.log.gz 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.1.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40001.log.gz 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40160.log 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39025.log 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39571.log 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39805.log.gz 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39025.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40160.log.gz 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41528.log 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40238.log 2024-03-29T11:22:40.158 INFO:teuthology.orchestra.run.smithi157.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39025.log.gz 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40355.log 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39571.log.gz 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39962.log 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40238.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41528.log.gz 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41098.log 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40238.log.gz 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr:gzip/var/log/ceph/ceph-client.admin.39962.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38555.log 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40355.log.gz 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41098.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39962.log.gz 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40277.log 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41098.log.gz 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38986.log 2024-03-29T11:22:40.159 INFO:teuthology.orchestra.run.smithi157.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38555.log.gz 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41606.log 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38986.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40277.log.gz 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41215.log 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41606.log: /var/log/ceph/ceph-client.admin.39259.log 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38986.log.gz 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41488.log 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41606.log.gz 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39259.log: 86.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39493.log 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41215.log.gz 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39259.log.gz 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41645.log 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40040.log 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39493.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41488.log.gz 2024-03-29T11:22:40.160 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39376.log 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39493.log.gz 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40040.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41645.log.gz 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41449.log 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40040.log.gz 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40433.log 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38791.log 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41449.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39376.log.gz 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40901.log 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41449.log.gz 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40940.log 86.6% 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40433.log.gz 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40901.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38791.log.gz 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40082.log 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:22:40.161 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40940.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40901.log.gz 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39766.log 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40940.log.gz 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- 86.0% /var/log/ceph/ceph-client.admin.38635.log 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40082.log.gz 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39610.log 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38635.log: gzip 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39766.log.gz 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39649.log 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39610.log: 85.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41137.log 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38635.log.gz 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39649.log: gzip 86.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39610.log.gz -- 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr: /var/log/ceph/ceph-client.admin.38830.log 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41137.log: -- /var/log/ceph/ceph-client.admin.39923.log 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39649.log.gz 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40823.log 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41137.log.gz 2024-03-29T11:22:40.162 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39923.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38830.log.gz 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39532.log 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39727.log 86.5% 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40823.log.gz 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41020.log 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39727.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39532.log.gz 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40784.log 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39727.log.gz 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41020.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41020.log.gz 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39844.log 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.40784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39142.log 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39844.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40784.log.gz 2024-03-29T11:22:40.163 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38360.log 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:gzip 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39844.log.gz 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39454.log 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38360.log: /var/log/ceph/ceph-client.admin.39142.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39454.log: 86.8% 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39142.log.gz 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38360.log.gz 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41059.log 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39454.log.gz 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.41059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38596.log 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.2.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.41059.log.gz 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40121.log 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.38596.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr: 85.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39415.log 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38596.log.gz 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41176.log 2024-03-29T11:22:40.164 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.39415.log: /var/log/ceph/ceph-client.admin.40121.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40121.log.gz 2024-03-29T11:22:40.165 INFO:teuthology.orchestra.run.smithi157.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39415.log.gz 2024-03-29T11:22:40.165 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40979.log 2024-03-29T11:22:40.165 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34929.log 2024-03-29T11:22:40.165 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.41176.log: /var/log/ceph/ceph-client.admin.40979.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41176.log.gz 2024-03-29T11:22:40.165 INFO:teuthology.orchestra.run.smithi157.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:22:40.165 INFO:teuthology.orchestra.run.smithi157.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40979.log.gz 2024-03-29T11:22:40.165 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.34929.log: /var/log/ceph/ceph-mon.c.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34929.log.gz 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34729.log 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34689.log 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34529.log 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.34689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34609.log 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34729.log.gz 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.34529.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34689.log.gz 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5/var/log/ceph/ceph-client.admin.34609.log: --verbose -- /var/log/ceph/ceph-client.admin.34809.log 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph.audit.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34529.log.gz 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34609.log.gz 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34569.log 2024-03-29T11:22:40.171 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.34809.log: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34809.log.gz 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.34569.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34649.log 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.7.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34569.log.gz 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.34649.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34649.log.gz 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34769.log 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.34769.log: 87.1%/var/log/ceph/ceph-mon.b.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34769.log.gz 2024-03-29T11:22:40.172 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:22:40.187 INFO:teuthology.orchestra.run.smithi157.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:22:51.483 INFO:teuthology.orchestra.run.smithi195.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:22:51.841 INFO:teuthology.orchestra.run.smithi195.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:22:51.912 INFO:teuthology.orchestra.run.smithi157.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:22:51.932 INFO:teuthology.orchestra.run.smithi157.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:22:55.841 INFO:teuthology.orchestra.run.smithi195.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:22:55.877 INFO:teuthology.orchestra.run.smithi195.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:22:55.966 INFO:teuthology.orchestra.run.smithi157.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:22:56.254 INFO:teuthology.orchestra.run.smithi157.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:23:29.790 INFO:teuthology.orchestra.run.smithi157.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:23:33.522 INFO:teuthology.orchestra.run.smithi157.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:23:33.524 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-29T11:23:33.524 INFO:teuthology.orchestra.run.smithi157.stderr:real 0m53.421s 2024-03-29T11:23:33.524 INFO:teuthology.orchestra.run.smithi157.stderr:user 2m32.737s 2024-03-29T11:23:33.524 INFO:teuthology.orchestra.run.smithi157.stderr:sys 0m5.444s 2024-03-29T11:23:46.574 INFO:teuthology.orchestra.run.smithi195.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:23:46.576 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-29T11:23:46.576 INFO:teuthology.orchestra.run.smithi195.stderr:real 1m6.433s 2024-03-29T11:23:46.576 INFO:teuthology.orchestra.run.smithi195.stderr:user 1m56.241s 2024-03-29T11:23:46.576 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m4.334s 2024-03-29T11:23:46.576 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:23:46.576 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676/remote/smithi157/log 2024-03-29T11:23:46.577 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:24:08.571 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676/remote/smithi195/log 2024-03-29T11:24:08.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:24:26.111 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:24:26.309 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=70c70d44256a48e0a710833b10c91e27 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:24:26.313 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:24:26.331 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:24:26.371 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:24:26.372 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:24:26.386 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:24:26.481 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:24:26.496 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:24:26.497 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:24:26.501 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:24:26.515 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:24:26.521 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:24:26.522 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:24:26.522 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-03-29T11:24:26.523 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +426us[ +430us] +/- 84ms 2024-03-29T11:24:26.523 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -152us[ -148us] +/- 79ms 2024-03-29T11:24:26.523 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 -244us[ -240us] +/- 79ms 2024-03-29T11:24:26.523 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:24:26.525 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:24:26.525 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-03-29T11:24:26.525 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +437us[ +437us] +/- 84ms 2024-03-29T11:24:26.525 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -139us[ -144us] +/- 79ms 2024-03-29T11:24:26.526 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -252us[ -252us] +/- 79ms 2024-03-29T11:24:26.526 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:24:26.528 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:24:26.548 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:24:26.550 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:24:26.569 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:24:26.641 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:24:26.714 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:24:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:24:26.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:24:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:24:26.779 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:24:26.788 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:24:26.803 INFO:teuthology.task.internal:Duration was 1340.995845 seconds 2024-03-29T11:24:26.803 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:24:26.818 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:24:26.818 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:24:26.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:24:26.865 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:24:26.866 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:24:27.197 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:24:27.198 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-03-29T11:24:27.198 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:24:27.219 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-03-29T11:24:27.219 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:24:27.247 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:24:27.248 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:24:27.262 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:24:27.316 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:24:27.316 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:24:27.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:24:27.454 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:24:27.465 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:24:27.465 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:24:27.492 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:24:27.519 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:24:27.532 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:24:27.535 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:24:27.558 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-03-29T11:24:27.585 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-03-29T11:24:27.611 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:24:27.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:24:27.634 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:24:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:24:27.666 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:24:27.678 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:24:27.679 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676/remote/smithi157 2024-03-29T11:24:27.679 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:24:27.717 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676/remote/smithi195 2024-03-29T11:24:27.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:24:27.752 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:24:27.752 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:24:27.755 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:24:27.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:24:27.813 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:24:27.813 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:24:27.828 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:24:27.828 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:24:27.830 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:24:27.843 INFO:teuthology.orchestra.run.smithi157.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:24 /home/ubuntu/cephtest 2024-03-29T11:24:27.857 INFO:teuthology.orchestra.run.smithi195.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:24 /home/ubuntu/cephtest 2024-03-29T11:24:27.858 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:24:27.870 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:24:27.903 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:24:27.930 DEBUG:teuthology.nuke:shortname: smithi157 2024-03-29T11:24:27.931 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:24:27.954 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:52:49.649025', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEzpVDeER3LSqzcPCX2onA/tmlitjWMAGbeI4lRfZ3+IlWdRVJ1SVubTqvC3OzjxhI6SW7/YDv4bE6CdsPrdO1s='} 2024-03-29T11:24:27.958 DEBUG:teuthology.nuke:shortname: smithi195 2024-03-29T11:24:27.958 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:24:27.978 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622676', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:52:49.650491', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKZbT9vO8FBP38Ohxdh8GmmMUosHlrzHzM47LuGueymclBqiORtuRYYEGfp1HyRqIjnEX2vis1T6Lk4LPFAsLGA='} 2024-03-29T11:24:28.007 INFO:teuthology.orchestra.console.smithi157:Power off 2024-03-29T11:24:28.008 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:24:28.034 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2024-03-29T11:24:28.034 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:24:28.050 INFO:teuthology.orchestra.console.smithi195:Power off 2024-03-29T11:24:28.050 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:24:28.059 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2024-03-29T11:24:28.071 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2024-03-29T11:24:28.071 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:24:28.093 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-03-29T11:24:32.061 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:24:32.094 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:24:32.172 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2024-03-29T11:24:32.206 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-03-29T11:24:36.175 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:24:36.207 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:24:36.285 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2024-03-29T11:24:36.317 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-03-29T11:24:36.386 INFO:teuthology.orchestra.console.smithi157:Power off completed 2024-03-29T11:24:36.418 INFO:teuthology.orchestra.console.smithi195:Power off completed 2024-03-29T11:24:36.522 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1340.9958448410034 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=70c70d44256a48e0a710833b10c91e27 status: fail success: false 2024-03-29T11:24:36.523 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:24:36.620 INFO:teuthology.run:FAIL